Foundations  and  Trends®  in 
Human-Computer  Interaction 
Vol.  1,  No.  3  (2007)  203-275 


now 


(c)  2007  M.  A.  Goodrich  and  A.  C.  Schultz 
DOI:  10.1561/1100000005 


the  essence  of  knowledge 


Human-Robot  Interaction:  A  Survey 

Michael  A.  Goodrich1  and  Alan  C.  Schultz2 

1  Brigham  Young  University,  Provo,  UT  84-602,  USA,  mike@cs.byu.edu 

2  US  Naval  Research  Laboratory,  Washington,  DC  20375,  USA, 
schultz@aic.nrl.  navy,  mil 

Abstract 

Human-Robot  Interaction  (HRI)  has  recently  received  considerable 
attention  in  the  academic  community,  in  labs,  in  technology  compa¬ 
nies,  and  through  the  media.  Because  of  this  attention,  it  is  desirable 
to  present  a  survey  of  HRI  to  serve  as  a  tutorial  to  people  outside 
the  field  and  to  promote  discussion  of  a  unified  vision  of  HRI  within 
the  field.  The  goal  of  this  review  is  to  present  a  unified  treatment  of 
HRI-related  problems,  to  identify  key  themes,  and  discuss  challenge 
problems  that  are  likely  to  shape  the  field  in  the  near  future.  Although 
the  review  follows  a  survey  structure,  the  goal  of  presenting  a  coher¬ 
ent  “story”  of  HRI  means  that  there  are  necessarily  some  well- written, 
intriguing,  and  influential  papers  that  are  not  referenced.  Instead  of 
trying  to  survey  every  paper,  we  describe  the  HRI  story  from  multiple 
perspectives  with  an  eye  toward  identifying  themes  that  cross  appli¬ 
cations.  The  survey  attempts  to  include  papers  that  represent  a  fair 
cross  section  of  the  universities,  government  efforts,  industry  labs,  and 
countries  that  contribute  to  HRI,  and  a  cross  section  of  the  disciplines 
that  contribute  to  the  field,  such  as  human,  factors,  robotics,  cognitive 
psychology,  and  design. 
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Introduction 


Human-Robot  Interaction  (HRI)  is  a  field  of  study  dedicated  to  under¬ 
standing,  designing,  and  evaluating  robotic  systems  for  use  by  or  with 
humans.  Interaction,  by  definition,  requires  communication  between 
robots  and  humans.  Communication  between  a  human  and  a  robot  may 
take  several  forms,  but  these  forms  are  largely  influenced  by  whether 
the  human  and  the  robot  are  in  close  proximity  to  each  other  or  not. 
Thus,  communication  and,  therefore,  interaction  can  be  separated  into 
two  general  categories: 

•  Remote  interaction  -  The  human  and  the  robot  are  not  co¬ 
located  and  are  separated  spatially  or  even  temporally  (for 
example,  the  Mars  Rovers  are  separated  from  earth  both  in 
space  and  time). 

•  Proximate  interaction  -  The  humans  and  the  robots  are  co¬ 
located  (for  example,  service  robots  may  be  in  the  same  room 
as  humans). 

Within  these  general  categories,  it  is  useful  to  distinguish  between 
applications  that  require  mobility,  physical  manipulation,  or  social 
interaction.  Remote  interaction  with  mobile  robots  is  often  referred 
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to  as  teleoperation  or  supervisory  control,  and  remote  interaction  with 
a  physical  manipulator  is  often  referred  to  as  telemanipulation.  Prox¬ 
imate  interaction  with  mobile  robots  may  take  the  form  of  a  robot 
assistant,  and  proximate  interaction  may  include  a  physical  interac¬ 
tion.  Social  interaction  includes  social,  emotive,  and  cognitive  aspects 
of  interaction.  In  social  interaction,  the  humans  and  robots  interact 
as  peers  or  companions.  Importantly,  social  interactions  with  robots 
appear  to  be  proximate  rather  than  remote.  Because  the  volume  of 
work  in  social  interactions  is  vast,  we  present  only  a  brief  survey;  a  more 
complete  survey  of  this  important  area  is  left  to  future  work. 

In  this  review,  we  present  a  survey  of  modern  HRI.  We  begin  by 
presenting  key  developments  in  HRI-related  fields  with  the  goal  of  iden¬ 
tifying  critical  technological  and  scientific  developments  that  have  made 
it  possible  for  HRI  to  develop  as  a  field  of  its  own.  We  argue  that  HRI  is 
not  simply  a  reframing  and  reformulation  of  previous  work,  but  rather 
a  new  field  of  scientific  study.  To  support  this  argument,  we  identify 
seminal  events  that  signal  the  emergence  of  HRI  as  a  field.  Although 
we  adopt  a  designer-centered  framing  of  the  review,  work  in  the  field 
requires  strong  interdisciplinary  blends  from  various  scientific  and  engi¬ 
neering  fields. 

After  surveying  key  aspects  in  the  emergence  of  HRI  as  a  field,  we 
define  the  HRI  problem  with  an  emphasis  on  those  factors  of  inter¬ 
action  that  a  designer  can  shape.  We  then  proceed  to  describe  the 
application  areas  that  drive  much  of  modern  HRI.  Many  of  these  prob¬ 
lems  are  extremely  challenging  and  have  strong  societal  implications. 
We  group  application  areas  into  the  previously  mentioned  two  general 
categories,  remote  and  proximate  interactions,  and  identify  important, 
influential,  or  thought-provoking  work  within  these  two  categories.  We 
follow  this  by  describing  common  solution  concepts  and  barrier  prob¬ 
lems  that  cross  application  domains  and  interaction  types.  We  then 
briefly  identify  related  work  from  other  fields  involving  humans  and 
machines  interacting,  and  summarize  the  review. 
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Early  History  of  Robotics  and 
Human-Machine-Interaction 


In  this  section,  we  briefly  survey  events  and  work  that  have  made  mod¬ 
ern  HRI  possible.  Clearly,  the  development  of  robots  was  the  essen¬ 
tial  first  step.  Although  robot  technology  was  primarily  developed 
in  the  mid  and  late  20th  century,  it  is  important  to  note  that  the 
notion  of  robot-like  behavior  and  its  implications  for  humans  have 
been  around  for  centuries  in  religion,  mythology,  philosophy,  and  fic¬ 
tion.  The  word  “robot”  originates  from  the  Czechoslovakian  word  rob- 
ota  which  means  work  [309].  “Robot”  appears  to  have  first  been  used 
in  Karel  Chapek’s  1920’s  play  Rossum’s  Universal  Robots ,  though  this 
was  by  no  means  the  earliest  example  of  a  human-like  machine.  Indeed, 
Leonardo  da  Vinci  sketched  a  mechanical  man  around  1495,  which 
has  been  evaluated  for  feasibility  in  modern  times  [250].  Pre-dating  da 
Vinci’s  humanoid  robot  are  automata  and  mechanical  creatures  from 
ancient  Egypt,  Greece,  and  China.  The  Iliad  refers  to  golden  maids 
that  behave  like  real  people  [125].  The  idea  of  golem ,  an  “artificial 
being  of  Hebrew  folklore  endowed  with  life”  has  been  around  for  cen¬ 
turies  [309]  and  was  discussed  by  Wiener  in  one  of  his  books  [315]. 
Ancient  Chinese  legends  and  compilations  mention  robot-like  creations, 
such  as  the  story  from  the  West  Zhou  Dynasty  (1066BC-771BC)  that 
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describes  how  the  craftsman  Yanshi  presented  a  humanoid.  The  cre¬ 
ation  looked  and  moved  so  much  like  a  human  that,  when  it  winked  at 
the  concubines,  it  was  necessary  to  dismantle  it  to  prove  that  it  was 
an  artificial  creation  [328].  Similar  robotic  devices,  such  as  a  wooden 
ox  and  floating  horse,  were  believed  to  have  been  invented  by  the  Chi¬ 
nese  strategist  Zhuge  Liang  [316],  and  a  famous  Chinese  carpenter  was 
reported  to  have  created  a  wooden/banrboo  magpie  that  could  stay 
aloft  for  up  to  three  days  [297].  More  recently,  robotic-like  automata, 
including  Vaucanson’s  duck,  have  been  created  [243].  Mechanical-like 
birds  were  present  in  the  1933  poem  Byzantium  by  W.  B.  Yeats  [326], 
and  robots  have  had  a  large  presence  in  science  fiction  literature,  most 
notably  Azimov’s  works  [12].  Indeed,  Asimov’s  Laws  of  Robotics  appear 
to  be  the  first  designer  guidelines  for  HRI. 

Early  robot  implementations  were  remotely  operated  devices  with 
no  or  minimal  autonomy  (Figure  2.1).  In  1898,  Nicola  Tesla  demon¬ 
strated  a  radio-controlled  boat,  which  he  described  as  incorporating 
“a  borrowed  mind.”  In  fact,  Tesla  controlled  the  boat  remotely.  His 
invention,  which  he  generalized  to  many  different  types  of  vehicles,  was 
described  in  patent  613,809,  “Method  and  Apparatus  for  Controlling 
Mechanism  of  Moving  Vessels.”  Tesla  hypothesized,  “. . .  you  see  there 


Fig.  2.1  Tesla’s  boat  [287]. 
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the  first  of  a  race  of  robots,  mechanical  men  which  will  do  the  laborious 
work  of  the  human  race.”  He  even  envisioned  one  or  more  operators 
simultaneously  directing  50  or  100  vehicles. 

Other  examples  include:  The  Naval  Research  Laboratory’s  “Elec¬ 
tric  Dog”  robot  from  1923,  attempts  to  remotely  pilot  bombers  during 
World  War  II,  the  creation  of  remotely  piloted  vehicles,  and  mechan¬ 
ical  creatures  designed  to  give  the  appearance  of  life.  As  technology 
evolved,  the  capabilities  of  remotely  operated  robots  have  grown  (see 
[95]  for  a  brief  history) .  This  is  perhaps  nowhere  more  evident  then  in 
the  very  successful  application  of  unmanned  underwater  vehicles  that 
have  been  used  to  explore  the  ocean’s  surface  to  find  lost  ships,  explore 
underwater  life,  assist  in  underwater  construction,  and  study  geother¬ 
mal  activity  [313]. 

Complementing  the  advances  in  robot  mechanics,  research  in  arti¬ 
ficial  intelligence  has  attempted  to  develop  fully  autonomous  robots. 
The  most  commonly  cited  example  of  an  early  autonomous  robot  was 
Shakey,  which  was  capable  of  navigating  through  a  block  world  under 
carefully  controlled  lighting  conditions  at  the  glacially  slow  speed  of 
approximately  2  meters  per  hour  [209].  Many  agree  that  these  early 
works  laid  a  foundation  for  much  that  goes  on  in  hybrid  control  archi¬ 
tectures  today  [196,  223]. 

A  breakthrough  in  autonomous  robot  technology  occurred  in  the 
mid  1980s  with  work  in  behavior-based  robotics  [10,  38].  Indeed,  it 
could  be  argued  that  this  work  is  a  foundation  for  many  current  robotic 
applications.  Behavior-based  robotics  breaks  with  the  monolithic  sense- 
plan-act  loop  of  a  centralized  system,  and  instead  uses  distributed 
sense-response  loops  to  generate  appropriate  responses  to  external  stim¬ 
uli.  The  combination  of  these  distributed  responses  produces  “emer¬ 
gent”  behavior  that  can  produce  very  sophisticated  responses  that 
are  robust  to  changes  in  the  environment.  A  second  important  break¬ 
through  for  autonomy  as  it  applies  to  HRI  is  the  emergence  of  hybrid 
architectures;  these  architectures  simultaneously  allow  sophisticated 
reactive  behaviors  that  provide  fundamental  robot  capabilities  along 
with  the  high-level  cognitive  reasoning  required  for  complex  and  endur¬ 
ing  interactions  with  humans.  Robot  behaviors  initially  focused  on 
mobility,  but  more  recent  contributions  seek  to  develop  lifelike  anthro- 
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pomorphic  behaviors  [323],  acceptable  behaviors  of  household  robots 
[158],  and  desirable  behaviors  for  robots  that  follow,  pass,  or  approach 
humans  [105,  220,  307]. 

The  development  of  robust  robot  platforms  and  communications 
technologies  for  extreme  environments  has  been  accomplished  by  NASA 
and  other  international  space  agencies.  Space  agencies  have  had  several 
high  profile  robotic  projects,  designed  with  an  eye  toward  safely  explor¬ 
ing  remote  planets  and  moons.  Examples  include  early  successes  of  the 
Soviet  Lunokhods  [95]  and  NASA’s  more  recent  success  of  exploring 
the  surface  of  Mars  [174,  317].  Importantly,  many  of  the  failures  have 
been  the  result  of  software  problems  rather  than  mechanical  failures. 
Complementing  NASA’s  fielded  robots  have  been  several  robots  devel¬ 
oped  and  evaluated  on  earth  [17].  Robonaut  is  a  well-known  example 
of  successful  teleoperation  of  a  humanoid  robot  [9],  and  this  work  is 
being  extended  at  a  rapid  pace  to  include  autonomous  movement  and 
reasoning.  Autonomous  robots  that  have  the  anthropomorphic  dimen¬ 
sions,  mimic  human-like  behaviors,  and  include  human-like  reasoning 
are  known  as  humanoid  robots;  work  in  this  area  has  been  ongoing  for 
over  a  decade  and  is  rapidly  expanding  [9,  23,  31,  37,  153,  273,  285]. 

Emerging  from  the  early  work  in  robotics,  human  factors  experts 
have  given  considerable  attention  to  two  paradigms  for  human-robot 
interaction:  teleoperation  and  supervisory  control.  At  the  teleopera¬ 
tion  extreme,  a  human  remotely  controls  a  mobile  robot  or  robotic 
arm.  With  supervisory  control,  a  human  supervises  the  behavior  of 
an  autonomous  system  and  intervenes  as  necessary.  Early  work  was 
usually  performed  by  people  who  were  interested  not  only  in  robotics 
but  also  factory  automation,  aviation,  and  intelligent  vehicles.  Work  in 
these  areas  is  typified  by  Sheridan’s  seminal  contributions  [267,  268], 
and  other  significant  contributions  from  human  factors  researchers 
[193,  314], 

Every  robot  application  appears  to  have  some  form  of  interaction, 
even  those  that  might  be  considered  “fully  autonomous.”  For  a  teleop- 
erated  robot,  the  type  of  interaction  is  obvious.  For  a  fully  autonomous 
robot,  the  interaction  may  consist  of  high-level  supervision  and  direc¬ 
tion  of  the  robot,  with  the  human  providing  goals  and  with  the  robot 
maintaining  knowledge  about  the  world,  the  task  and  its  constraints. 
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In  addition,  the  interactions  may  be  through  observation  of  the  environ¬ 
ment  and  implicit  communications  by,  for  example,  the  robot  respond¬ 
ing  to  what  its  human  peer  is  doing.  Taking  a  very  broad  and  general 
view  of  HRI,  one  might  consider  that  it  includes  developing  algorithms, 
programming,  testing,  refining,  fielding,  and  maintaining  the  robots. 
In  this  case,  interaction  consists  primarily  in  discovering  and  diag¬ 
nosing  problems,  solving  these  problems,  and  then  reprogramming  (or 
rewiring)  the  robot.  The  difference  between  this  type  of  “programming- 
based”  interaction  and  modern  HRI  is  that  the  field  currently  empha¬ 
sizes  efficient  and  dynamic  interactions  rather  than  just  infrequent 
interactions.  However,  some  researchers  are  addressing  programming- 
based  of  interaction  by  exploring  efficient  programming  paradigms  to 
support  robot  development  [128,  327]. 
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Emergence  of  HRI  as  a  Field 


Although  there  is  much  work  that  can  be  considered  HRI,  the  multi¬ 
disciplinary  field  started  to  emerge  in  the  mid  1990s  and  early  years  of 
2000.  Key  numerous  events  occurred  in  this  time  frame,  with  the  main 
catalyst  being  a  multi-disciplinary  approach;  researchers  from  robotics, 
cognitive  science,  human  factors,  natural  language,  psychology,  and 
human-computer  interaction  started  to  come  together  at  these  events 
specifically  recognizing  the  importance  of  working  together. 

The  earliest  scientific  meeting,  which  started  in  1992  and  continues 
annually,  is  the  IEEE  International  Symposium  on  Robot  &  Human 
Interactive  Communication  (RoMan).  Although  recently  this  confer¬ 
ence  has  attracted  a  more  multi-disciplinary  research  community,  his¬ 
torically  it  has  been  heavily  dominated  by  the  robotics  discipline.  In 
2000,  the  IEEE/Robotics  Society  of  Japan  created  the  International 
Conference  on  Humanoid  Robots  which  highlights  anthropomorphic 
robots  and  robotic  behaviors. 

From  the  late  1990s  until  recently,  there  have  been  many  work¬ 
shops  and  conference  tracks  dedicated  to  HRI,  including  ones  associated 
with  the  Association  for  the  Advancement  of  Artificial  Intelli¬ 
gence’s  (AAAI)  Symposia  Series,  IEEE  International  Conference  on 
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Robotics  and  Automation  (ICRA),  Robotics  Systems  and  Sciences,  the 
IEEE/Robotics  Society  of  Japan  International  Conference  on  Intelli¬ 
gent  Robot  and  Systems,  among  others,  and  the  annual  meeting  of  the 
Human  Factors  and  Ergonomics  Society. 

In  2001,  the  US  National  Science  Foundation  and  Defense  Advanced 
Research  Projects  Agency  sponsored  a  workshop  on  human-robot  inter¬ 
action,  organized  by  Dr  Robin  Murphy  and  Dr  Erica  Rogers  [46].  The 
purpose  of  this  workshop  was  to  bring  together  a  highly  multidisci¬ 
plinary  group  of  researchers  working  in  areas  close  to  HRI,  and  to 
help  identify  the  issues  and  challenges  in  HRI  research.  Although  much 
research  had  been  done  prior  to  this  event,  some  consider  it  to  be  sem¬ 
inal  in  the  emergence  of  the  field  as  its  own  discipline.  A  second  NSF 
workshop  was  held  in  2006  [181]. 

In  July  2004,  IEEE-RAS  and  the  International  Foundation  of 
Robotics  Research  (IFRR)  sponsored  a  summer  school  on  “Human- 
Robot  Interaction.”  This  event  brought  together  six  experts  from 
the  field  of  HRI  and  approximately  30  PhD  students  for  a  week  in 
Volterra  Italy  for  four  intensive  days  of  lectures  and  events.  A  sim¬ 
ilar  event  that  has  been  held  annually  since  2004  is  the  Rescue 
Robotics  Camp  (see,  for  example,  [239,  240]).  About  the  same  time, 
a  series  of  special  issues  dedicated  to  HRI  began  to  appear  in  journals 
[5,  157,  171,  201,  261]. 

In  2005,  the  US  National  Research  Council  sponsored  a  work¬ 
shop  entitled  “Interfaces  for  Ground  and  Air  Military  Robots”  [64]. 
The  workshop  discussed  emerging  interface  and  autonomy  themes  that 
could  be  used  across  multiple  scales  to  support  primarily  remote  inter¬ 
action  of  humans  and  robots. 

The  Japan  Association  for  the  2005  World  Exposition  conducted 
a  Robot  Project  at  EXPO  2005  that  featured  a  wide  range  of  robots 
[6].  Guide,  cleaning,  service,  and  assistive  robots  were  among  the  many 
robots  that  were  featured. 

Starting  in  2006,  the  ACM  International  Conference  on  Human- 
Robot  Interaction  was  created  to  specifically  address  the  multidis¬ 
ciplinary  aspects  of  HRI  research.  Reflecting  this  multidisciplinary 
nature,  the  2007  conference  was  co-sponsored  by  the  ACM  Special 
Interest  Group  on  Computer  Human  Interaction,  the  ACM  Special 
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Interest  Group  on  Artificial  Intelligence,  and  the  IEEE  Robotics  and 
Automation  Society  (RAS),  with  co-technical  sponsorship  from  AAAI, 
the  Human  Factors  and  Ergonomics  Society,  and  the  IEEE  Systems, 
Man,  and  Cybernetics  Society.  Associated  with  the  HRI  conference  is 
a  NSF-funded  student  workshop.  Other  conferences  have  had  a  strong 
interest  in  HRI  including  the  following:  the  Humanoid  Robotics  work¬ 
shops;  the  IEEE  International  Workshop  on  Safety,  Security,  and  Res¬ 
cue  Robotics;  and  the  Performance  Metrics  for  Intelligent  Systems 
workshop. 

In  2006,  the  European  Land-Robot  Trial  (ELROB)  was  created  to 
“provide  an  overview  of  the  European  state-of-the-art  in  the  field  of 
[Unmanned  Ground  Vehicles]”  [87].  Such  systems  frequently  included 
robust  user  interfaces  intended  for  field  conditions  in  challenging 
environments,  such  as  those  faced  in  military  and  first  responder 
domains. 

Another  big  influence  in  the  emergence  of  HRI  has  been  compe¬ 
titions.  The  two  with  the  greatest  impact  have  been  (a)  the  AAAI 
Robotics  Competition  and  Exhibition  and  (b)  the  Robocup  Search  and 
Rescue  competition.  The  Sixth  AAAI  Robot  Competition  in  1997  had 
the  first  competition  specifically  designed  for  HRI  research  called  “Hors 
d’ Oeuvres  Anyone?”  The  goal  of  the  competition  was  for  a  robot  to 
serve  snacks  to  attendees  of  the  conference  during  the  conference  recep¬ 
tion.  This  event  was  repeated  in  1998.  Starting  in  1999,  a  new  grand 
challenge  event  was  introduced.  For  this  competition,  a  team’s  robot 
had  to  be  dropped  off  at  the  front  door  of  the  conference  venue  and, 
through  interaction  with  people,  find  its  way  to  the  registration  desk, 
register  for  the  conference,  and  then  find  its  way  at  the  correct  time 
to  a  place  where  it  was  to  give  a  presentation.  This  task  was  designed 
to  be  hard  enough  to  take  many  years  to  accomplish,  helping  to  drive 
research  (see,  for  example,  [276]).  In  recent  years  this  conference  held 
several  general  human- interaction  events. 

In  some  cases,  an  application  domain  has  helped  to  draw  the 
field  together.  Three  very  influential  areas  are  robot-assisted  search 
and  rescue,  assistive  robots,  and  space  exploration.  Literature  from 
each  of  these  domains  is  addressed  further  in  a  subsequent  section. 
Robot-assisted  search  and  rescue  has  been  a  domain  in  which  the 
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robotics  field  has  worked  directly  with  the  end  users  which,  in  this 
case,  consists  of  specially  trained  rescue  personnel.  The  typical  search 
and  rescue  situation  involves  using  a  small  robot  to  enter  into  a  poten¬ 
tially  dangerous  rubble  pile  to  search  for  victims  of  a  building  col¬ 
lapse.  The  robots  are  typically  equipped  with  a  video  camera  and 
possibly  chemical  and  temperature  sensors,  and  may  sometimes  be 
equipped  with  a  manipulator  with  which  they  can  alter  the  envi¬ 
ronment.  The  goal  is  to  quickly  survey  an  area  that  would  other¬ 
wise  be  unsafe  for  a  human  searcher  to  enter,  and  gather  information 
about  victim  location  and  structural  stability.  Because  of  the  inher¬ 
ently  unstructured  nature  of  search  and  rescue  domains,  the  interac¬ 
tions  between  the  human  and  the  robot  are  very  rich.  Consequently, 
many  HRI  issues  are  addressed  in  the  problem,  and  several  ongoing 
competitions  are  held  to  encourage  robotics  researchers  to  participate 
[159,  199,  325]. 

Assistive  robot  systems  seek  to  provide  physical,  mental,  or  social 
support  to  persons  who  could  benefit  from  it  such  as  the  elderly  or 
disabled.  Assistive  robotics  is  important  to  HRI  because  it  empha¬ 
sizes  proximate  interaction  with  potentially  disabled  persons.  HRI 
challenges  from  this  domain  include  providing  safe  physical  contact 
or  moving  within  very  close  proximity.  The  challenges  also  include 
supporting  effective  social  interactions  through  cognitive  and  emo¬ 
tive  computing,  and  through  natural  interactions  such  as  gesture  and 
speech.  Although  sometimes  referred  to  by  names  other  than  robots,  the 
types  of  robots/machines  used  in  assistive  applications  vary  widely  in 
their  physical  appearance,  and  include  wheelchairs,  mobile  robots  with 
manipulators,  animal-like  robots,  and  humanoids  [90,  206,  246,  299]. 
Because  of  the  close  proximity  and  sometimes  long-term  interactions, 
appropriate  HRI  in  assistive  robotics  may  be  sensitive  to  cultural  influ¬ 
ences  [152,  270]. 

Space  robotics  has  also  been  an  important  domain  for  HRI  because 
of  the  challenges  that  arise  under  such  extreme  operating  conditions. 
These  challenges  include  operating  a  remote  robot  when  the  time  lag 
can  be  a  significant  factor,  or  interacting  in  close  proximity  such  as 
when  a  robot  assistant  helps  an  astronaut  in  exploring  the  surface 
of  a  planetary  body.  A  typical  anticipated  situation  is  a  geological 
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study  that  involves  prolonged  work  on  the  surface  of  a  planetary  body, 
possibly  using  specialized  sensors  such  as  ground-penetrating  radar  and 
specialized  manipulators  such  as  a  drill  and  hammer  [84].  Information 
gathered  by  the  robot  needs  to  be  returned  either  (a)  to  an  astronaut 
who  is  co-located  with  the  robot  or  (b)  to  a  ground-bases  science  team 
who  then  form  real-time  hypotheses  that  are  used  to  modify  the  behav¬ 
ior  of  the  robot. 


4 


What  Defines  an  HRI  Problem? 


The  HRI  problem  is  to  understand  and  shape  the  interactions  between 
one  or  more  humans  and  one  or  more  robots.  Interactions  between 
humans  and  robots  are  inherently  present  in  all  of  robotics,  even  for 
so  called  autonomous  robots  —  after  all,  robots  are  still  used  by  and 
are  doing  work  for  humans.  As  a  result,  evaluating  the  capabilities  of 
humans  and  robots,  and  designing  the  technologies  and  training  that 
produce  desirable  interactions  are  essential  components  of  HRI.  Such 
work  is  inherently  interdisciplinary  in  nature,  requiring  contributions 
from  cognitive  science,  linguistics,  and  psychology;  from  engineering, 
mathematics,  and  computer  science;  and  from  human  factors  engineer¬ 
ing  and  design. 

Although  analysis  of  anticipated  and  existing  interaction  patterns 
is  essential,  it  is  helpful  to  adopt  the  designer’s  perspective  by  breaking 
the  HRI  problem  into  its  constituent  parts.  In  essence,  a  designer  can 
affect  five  attributes  that  affect  the  interactions  between  humans  and 
robots: 


•  Level  and  behavior  of  autonomy, 

•  Nature  of  information  exchange, 

•  Structure  of  the  team, 
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•  Adaptation,  learning,  and  training  of  people  and  the 
robot,  and 

•  Shape  of  the  task. 

Interaction,  the  process  of  working  together  to  accomplish  a  goal, 
emerges  from  the  confluence  of  these  factors.  The  designer  attempts 
to  understand  and  shape  the  interaction  itself,  with  the  objective  of 
making  the  exchange  between  humans  and  robots  beneficial  in  some 
sense.  We  now  discuss  each  of  these  attributes  in  detail,  including  ref¬ 
erences  from  the  literature. 

4.1  Autonomy 

Designing  autonomy  consists  of  mapping  inputs  from  the  environment 
into  actuator  movements,  representational  schemas,  or  speech  acts. 
There  are  numerous  formal  definitions  of  autonomy  and  intelligence  in 
the  literature  [7,  20,  119,  184,  256],  many  of  which  arise  in  discussions 
of  adjustable  or  dynamic  autonomy  [30].  One  operational  characteriza¬ 
tion  of  autonomy  that  applies  to  mobile  robots  is  the  amount  of  time 
that  a  robot  can  be  neglected,  or  the  neglect  tolerance  of  the  robot  [68]. 
A  system  with  a  high  level  of  autonomy  is  one  that  can  be  neglected 
for  a  long  period  of  time  without  interaction.  However,  this  notion  of 
autonomy  does  not  encompass  Turing-type  notions  of  intelligence  that 
might  be  more  applicable  to  representational  or  speech-act  aspects  of 
autonomy. 

Autonomy  is  not  an  end  in  itself  in  the  field  of  HRI,  but  rather  a 
means  to  supporting  productive  interaction.  Indeed,  autonomy  is  only 
useful  insofar  as  it  supports  beneficial  interaction  between  a  human  and 
a  robot.  Consequently,  the  physical  embodiment  and  type  of  autonomy 
varies  dramatically  across  robot  platforms;  see  Figure  4.1,  which  shows 
a  cross  section  of  the  very  many  different  types  of  physical  robots. 

Perhaps  the  most  strongly  human-centered  application  of  the  con¬ 
cept  of  autonomy  is  in  the  notion  of  level  of  autonomy  ( LOA ).  Levels  of 
autonomy  describe  to  what  degree  the  robot  can  act  on  its  own  accord. 
Although  many  descriptions  of  LOA  have  been  seen  in  the  literature, 
the  most  widely  cited  description  is  by  Tom  Sheridan  [269].  In  Sheri¬ 
dan’s  scale,  there  is  a  continuum  from  the  entity  being  completely  con- 
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Fig.  4.1  Representative  types  of  robots.  In  clockwise  order  beginning  in  the  upper  left: 
RepileeQ2  —  an  extremely  sophisticated  humanoid  [136];  Robota  —  humanoid  robots  as 
“educational  toys”  [21];  SonyAIBO  —  a  popular  robot  dog  ;  (below  the  AIBO)  A  sophisti¬ 
cated  unmanned  underwater  vehicle  [176];  Shakey  —  one  of  the  first  modern  robots,  cour¬ 
tesy  of  SRI  International,  Menlo  Park,  CA  [279] ;  Kismet  —  an  anthropomorphic  robot  with 
exaggerated  emotion  [65];  Raven  —  a  mini-UAV  used  by  the  US  military  [186];  iCAT  — 
an  emotive  robot  [REF];  iRobot®  PackBot®  —  a  robust  ground  robot  used  in  military 
applications  [135].  (All  images  used  with  permission.) 


trolled  by  a  human  (i.e. ,  teleoperated) ,  through  the  entity  being  com¬ 
pletely  autonomous  and  not  requiring  input  or  approval  of  its  actions 
from  a  human  before  taking  actions: 


1.  Computer  offers  no  assistance;  human  does  it  all. 

2.  Computer  offers  a  complete  set  of  action  alternatives. 

3.  Computer  narrows  the  selection  down  to  a  few  choices. 

4.  Computer  suggests  a  single  action. 

5.  Computer  executes  that  action  if  human  approves. 

6.  Computer  allows  the  human  limited  time  to  veto  before  auto¬ 
matic  execution. 

7.  Computer  executes  automatically  then  necessarily  informs 
the  human. 

8.  Computer  informs  human  after  automatic  execution  only  if 
human  asks. 
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9.  Computer  informs  human  after  automatic  execution  only  if 
it  decides  too. 

10.  Computer  decides  everything  and  acts  autonomously,  ignor¬ 
ing  the  human. 

Variations  of  this  scale  have  been  developed  and  used  by  various  authors 
[144,  222],  Importantly,  Miller  and  Parasuraman  have  noted  that  such 
scales  may  not  be  applicable  to  an  entire  problem  domain  but  are 
rather  most  useful  when  applied  to  each  subtask  within  a  problem 
domain  [188].  The  authors  further  suggest  that  previous  scales  actually 
represent  an  average  over  all  tasks. 

While  such  (average)  scales  are  appropriate  to  describe  how 
autonomous  a  robot  is,  from  a  human-robot  interaction  point  of  view, 
a  complementary  way  to  consider  autonomy  is  by  describing  to  what 
level  the  human  and  robot  interact  and  the  degree  to  which  each  is  capa¬ 
ble  of  autonomy.  The  scale  presented  in  Figure  4.2  gives  an  emphasis 
to  mixed-initiative  interaction,  which  has  been  defined  as  a  “flexible 
interaction  strategy  in  which  each  agent  (human  and  [robot])  con¬ 
tributes  what  it  is  best  suited  at  the  most  appropriate  time”  [122], 
Various  and  different  HRI  issues  arise  along  this  scale.  On  the  direct 
control  side,  the  issues  tend  toward  making  a  user  interface  that  reduces 
the  cognitive  load  of  the  operator.  On  the  other  extreme  of  peer- 
to-peer  collaboration,  issues  arise  in  how  to  create  robots  with  the 
appropriate  cognitive  skills  to  interact  naturally  or  efficiently  with 
a  human. 

Note  that  in  order  for  the  robot  to  achieve  peer-to-peer  collabo¬ 
ration,  it  must  indeed  be  able  to  flexibly  exhibit  “full  autonomy”  at 
appropriate  times.  Moreover,  it  may  need  to  support  social  interactions. 


,.OV 
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direct  control 


dynamic  autonomy 


Fig.  4.2  Levels  of  autonomy  with  emphasis  on  human  interaction. 


220  What  Defines  an  HRI  Problem? 


As  a  result,  peer-to-peer  collaboration  may  be  considered  more  difficult 
to  achieve  than  full  autonomy. 

Autonomy  is  implemented  using  techniques  from  control  theory, 
artificial  intelligence,  signal  processing,  cognitive  science,  and  linguis¬ 
tics.  A  common  autonomy  approach  is  sometimes  referred  to  as  the 
sens  e-plan- act  model  of  decision-making  [196].  This  model  has  been  a 
target  of  criticism  [39]  and  sometimes  rightfully  so,  but  much  of  the 
criticism  may  be  a  function  of  the  early  capacities  of  robots  such  as 
Shakey  [209]  rather  than  failings  of  the  model  per  se.  This  model  is 
typified  by  artificial  intelligence  techniques,  such  as  logics  and  planning 
algorithms  [253].  The  model  can  also  incorporate  control  theoretic  con¬ 
cepts,  which  have  been  used  very  successfully  in  aviation,  aeronautics, 
missile  control,  and  etc.  (see,  for  example,  [175]). 

In  the  mid  1980s,  Brooks,  Arkin,  and  others  revolutionized  the  field 
of  robotics  by  introducing  a  new  autonomy  paradigm  that  came  to  be 
known  as  behavior-based  robotics.  In  this  paradigm,  behavior  is  gen¬ 
erated  from  a  set  of  carefully  designed  autonomy  modules  that  are 
then  integrated  to  create  an  emergent  system  [10,  38,  40].  These  mod¬ 
ules  generate  reactive  behaviors  that  map  sensors  directly  to  actions, 
sometimes  with  no  intervening  internal  representations.  This  model  for 
behavior  generation  was  accompanied  by  hardware  development  that 
allowed  autonomy  modules  to  be  implemented  in  the  small  form  factors 
required  for  many  robotics  applications. 

Today,  many  researchers  build  sense-think-act  models  on  top  of  a 
behavior-based  substrate  to  create  hybrid  architectures  [196].  In  these 
systems,  the  low-level  reactivity  is  separated  from  higher  level  reason¬ 
ing  about  plans  and  goals  [28] .  Some  have  developed  mathematics  and 
frameworks  that  can  be  viewed  as  formalizations  of  hybrid  architectures 
and  which  are  referred  to  as  theories  of  intelligent  control  [7,  255].  Inter¬ 
estingly,  some  of  the  most  challenging  problems  in  developing  (hybrid) 
behaviors  is  in  producing  natural  and  robust  activity  for  a  humanoid 
robot  [194,  273,  323]. 

Complementing  the  advancement  of  robotic  control  algorithms  has 
been  the  advancement  of  sensors,  sensor-processing,  and  reasoning  algo¬ 
rithms.  This  is  best  represented  by  the  success  of  the  field  of  prob¬ 
abilistic  robotics ,  typified  by  probabilistic  algorithms  for  localization 
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and  mapping  [161,  289].  It  is  no  overstatement  to  say  that  these 
algorithms,  which  frequently  exploit  data  from  laser  and  other  range 
finder  devices,  have  allowed  autonomy  to  become  truly  useful  for 
mobile  robots  [290],  especially  those  that  require  remote  interaction 
through  periods  of  autonomous  behavior  and  autonomous  path  plan¬ 
ning  [42,  276,  284,  291,  293].  Although  probabilistic  algorithms  can  be 
computationally  expensive,  the  memory  capacity,  computational  speed, 
and  form  factor  of  modern  computers  have  allowed  these  algorithms  to 
be  deployable. 

The  areas  of  representing  knowledge  and  performing  reasoning, 
especially  in  team  contexts,  have  also  grown.  Example  developments 
include  the  emergence  of  belief-desire-intention  architectures  [321], 
joint  intention  theory  [60],  affect-based  computing  [31,  223,  229],  and 
temporal  logics. 

4.2  Information  Exchange 

Autonomy  is  only  one  of  the  components  required  to  make  an  inter¬ 
action  beneficial.  A  second  component  is  the  manner  in  which  infor¬ 
mation  is  exchanged  between  the  human  and  the  robot  (Figure  4.3). 
Measures  of  the  efficiency  of  an  interaction  include  the  interaction 
time  required  for  intent  and/or  instructions  to  be  communicated  to 
the  robot  [68],  the  cognitive  or  mental  workload  of  an  interaction  [268], 
the  amount  of  situation  awareness  produced  by  the  interaction  [88]  (or 
reduced  because  of  interruptions  from  the  robot),  and  the  amount  of 
shared  understanding  or  common  ground  between  humans  and  robots 
[143,  160]. 

There  are  two  primary  dimensions  that  determine  the  way  informa¬ 
tion  is  exchanged  between  a  human  and  a  robot:  the  communications 
medium  and  the  format  of  the  communications.  The  primary  media  are 
delineated  by  three  of  the  five  senses:  seeing,  hearing,  and  touch.  These 
media  are  manifested  in  HRI  as  follows: 

•  visual  displays,  typically  presented  as  graphical  user  inter¬ 
faces  or  augmented  reality  interfaces  [15,  145,  154,  208], 

•  gestures,  including  hand  and  facial  movements  and  by 
movement-based  signaling  of  intent  [31,  73,  247,  305], 
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Fig.  4.3  Types  of  human-robot  interaction.  Counterclockwise  from  top  left:  haptic  robot 
interaction  from  Georgia  Tech  [102],  a  “physical  icon”  for  flying  a  UAV  from  Brigham 
Young  University,  peer-to-peer  interaction  with  the  robot  Kaspar  from  the  University 
of  Hertfordshire  [298],  teleoperation  of  NASA’s  Robonaut  [205],  a  PDA-based  interface 
for  flying  a  UAV  from  Brigham  Young  University,  gesture-  and  speech-based  interac¬ 
tion  with  MIT’s  Leonardo  [189],  a  touchscreen  interaction  with  a  Cogniron  robot  [169], 
and  (center)  physical  interaction  with  the  RI-MAN  robot  [24].  (All  images  used  with 
permission.) 


•  speech  and  natural  language,  which  include  both  audi¬ 
tory  speech  and  text-based  responses,  and  which  fre¬ 
quently  emphasize  dialog  and  mixed-initiative  interaction 
[126,  227], 

•  non-speech  audio,  frequently  used  in  alerting  [78] ,  and 

•  physical  interaction  and  haptics,  frequently  used  remotely 
in  augmented  reality  or  in  teleoperation  to  invoke  a  sense 
of  presence  especially  in  telemanipulation  tasks  [10,  282], 
and  also  frequently  used  proximately  to  promote  emotional, 
social,  and  assistive  exchanges  [56,  124,  172,  272], 
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Recently,  attention  has  focused  on  building  multimodal  interfaces  [226], 
partly  motivated  by  a  quest  to  reduce  workload  in  accordance  to  Wick- 
ens’  multiple  resource  theory  [314]  and  partly  motivated  by  a  desire  to 
make  interactions  more  natural  and  easier  to  learn  [248,  254,  281]. 

The  format  of  the  information  exchange  varies  widely  across 
domains.  Speech-  and  natural  language-based  exchanges  can  be 
scripted  and  based  on  a  formal  language,  can  attempt  to  support  full 
natural  language,  or  can  restrict  natural  language  to  a  subset  of  lan¬ 
guage  and  a  restricted  domain  (see,  for  example,  [52,  120,  251,  275, 
276]).  Importantly,  speech-based  exchanges  must  not  only  address  the 
content  of  information  exchanged,  but  also  the  rules  of  such  exchange 
a  la  the  Gricean  maxims  [118],  which  ask  to  what  extent  the  speech 
is  truthful,  relevant,  clear,  and  informative.  Haptic  information  pre¬ 
sentation  can  include  giving  warnings  through  vibrations,  promoting 
the  feeling  of  telepresence,  supporting  spatial  awareness  through  hap¬ 
tic  vests,  and  communicating  specific  pieces  of  information  through 
haptic  icons  (see,  for  example,  [53,  164,  235]).  Audio  information 
presentation  can  include  auditory  alerts,  speech-based  information 
exchange,  and  3D  awareness  (see,  for  example,  [285]).  Presenting  social 
information  can  include  attentional  cueing,  gestures,  sharing  physical 
space,  imitation,  sounds,  facial  expression,  speech  and  natural  language 
[22,  35,  36,  83,  94,  203,  149,  260].  Finally,  graphical  user  interfaces 
present  information  in  ways  that  include  ecological  displays,  immer¬ 
sive  virtual  reality,  and  traditional  windows- type  interactions  (see,  for 
example,  [9,  15,  185,  208]). 

4.3  Teams 

HRI  problems  are  not  restricted  to  a  single  human  and  a  single  robot, 
though  this  is  certainly  one  important  type  of  interaction.  Robots  used 
in  search  and  rescue,  for  example,  are  typically  managed  by  two  or  more 
people,  each  with  special  roles  in  the  team  [197,  264].  Similarly,  manag¬ 
ing  Unmanned/Uninhabited  Air  Vehicles  (UAVs)  is  typically  performed 
by  at  least  two  people:  a  “pilot,”  who  is  responsible  for  navigation  and 
control,  and  a  sensor/payload  operator,  who  is  responsible  for  manag¬ 
ing  cameras,  sensors,  and  other  payloads  [82,  182], 
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A  question  that  has  received  considerable  attention,  but  which  is 
directly  addressed  by  few  scientific  studies,  is  how  many  remote  robots 
a  single  human  can  manage.  In  general,  the  answer  is  dependent  on 
factors  such  as  the  level  of  autonomy  of  the  robot  (e.g.,  teleoperation 
requires  a  great  deal  of  direct  attention  from  the  human),  the  task 
(which  defines  not  only  the  type  and  quantity  of  data  being  returned 
to  the  human  but  also  the  amount  of  attention  and  cognitive  load 
required  of  the  human),  and  the  available  modes  of  communication. 

In  the  search  and  rescue  domain,  Murphy  [197]  asserts  that  the 
demands  of  the  task,  the  form  factor  of  the  robot,  and  the  need  to  pro¬ 
tect  robot  operators  requires  at  least  two  operators,  an  observation  that 
has  received  strong  support  from  field  trials  using  mature  technologies 
[45],  and  partial  support  in  search  and  rescue  competitions  using  less 
mature  but  more  ambitious  technologies  [264].  In  other  domains,  some 
assert  that,  given  sophisticated  enough  autonomy  and  possibly  coordi¬ 
nated  control,  it  is  possible  for  a  single  human  to  manage  more  than  one 
robotic  asset  [187,  190]  though  the  task  may  still  need  another  human 
to  interpret  sensor  information.  Still  others  assert  that  this  problem  is 
ill-formed  when  robots  are  used  primarily  as  an  information-gathering 
tool  [121].  An  intermediate  position  is  that  the  right  question  should 
not  focus  on  how  many  robots  can  be  managed  by  a  single  human,  but 
rather  the  following:  how  many  humans  does  it  take  to  efficiently  man¬ 
age  a  fixed  number  of  robots,  allowing  for  the  possibility  of  adaptable 
autonomy  and  dynamic  handoffs  between  humans  [266]. 

One  measure  that  has  received  some  attention  in  the  literature  is 
the  notion  of  fan-out,  which  represents  an  upper  bound  on  the  number 
of  independent,  homogeneous  robots  that  a  single  person  can  manage 
[216,  217].  This  measure  is  supported  by  a  limited  set  of  techniques 
for  estimating  it  [68].  Some  work  has  been  done  to  refine  the  fan-out 
to  apply  to  teams  of  heterogeneous  robots  [112]  and  to  tighten  the 
bound  by  identifying  various  aspects  of  interaction  [190].  In  its  present 
form,  however,  it  is  clear  that  fan-out  is  only  a  designer  guideline  and  is 
insufficient,  for  example,  to  provide  a  trigger  strategy  [144]  for  adaptive 
automation.  Alternatives  to  fan-out  include  predicting  the  performance 
of  a  team  of  heterogeneous  robots  from  measurements  of  neglect  toler¬ 
ance  and  interaction  times  [69]. 
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In  addition  to  the  number  of  humans  and  robots  in  a  team,  a  key 
problem  is  the  organization  of  the  team  [98,  213].  One  important  orga¬ 
nizational  question  is  who  has  the  authority  to  make  certain  decisions: 
robot,  interface  software,  or  human?  Another  important  question  is 
who  has  the  authority  to  issue  instructions  or  commands  to  the  robot 
and  at  what  level:  strategic,  tactical,  or  operational?  A  third  important 
question  is  how  conflicts  are  resolved,  especially  when  robots  are  placed 
in  peer-like  relationships  with  multiple  humans.  A  fourth  question  is 
how  roles  are  defined  and  supported:  is  the  robot  a  peer,  an  assistant, 
or  a  slave;  does  it  report  to  another  robot,  to  a  human,  or  is  it  fully 
independent? 

Spanning  all  of  these  questions  is  whether  the  organizational  struc¬ 
ture  is  static  or  dynamic,  with  changes  in  responsibilities,  authorities, 
and  roles.  In  one  study,  managing  multiple  robots  in  a  search  and  rescue 
domain  under  either  manual  or  coordinated  control  produced  results 
that  strongly  favored  coordinated  control  [308].  In  another  study,  four 
autonomy  configurations,  including  two  variations  of  sliding  autonomy, 
were  managed  by  a  human  working  on  a  construction  task  with  a  team 
of  heterogeneous  robots  [266] .  In  this  study,  the  tradeoffs  between  time 
to  completion,  quality  of  behavior,  and  operator  workload  were  strongly 
evident.  This  result  emphasizes  the  importance  of  using  dynamic  auton¬ 
omy  when  the  world  is  complex  and  varies  over  time.  In  a  third  study, 
researchers  explored  how  making  coordination  between  robots  explicit 
can  reduce  failures  and  improve  consistency,  in  contrast  to  traditional 
interfaces  [147].  In  a  fourth  study,  researchers  explored  the  minimal 
amount  of  gestural  information  required  to  command  various  forma¬ 
tions  to  a  team  of  robots  [277]. 

In  many  existing  and  envisioned  problems,  HRI  will  include  not  only 
humans  and  robots  interacting  with  each  other,  but  also  coordinating 
with  software  agents.  The  most  simple  form  of  this  is  a  three-agent 
problem  which  occurs  when  an  intelligent  interface  is  the  intermediary 
between  a  human  and  a  remote  robot  [249].  In  this  problem,  the  inter¬ 
face  agent  can  monitor  and  categorize  human  behavior,  monitor  and 
detect  problems  with  the  robot,  and  support  the  human  when  workload 
levels,  environment  conditions,  and  robot  capabilities  change.  A  more 
complicated  form  of  this  teaming  is  in  anticipated  NASA  applications 
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where  multiple  distributed  humans  will  interact  with  robots  and  with 
software  agents  that  coordinate  mission  plans,  human  activities,  and 
system  resources  [29] . 

A  final  issue  that  is  starting  to  gain  attention  is  the  role  of  the 
human  [262].  While  much  of  the  discussion  up  to  this  point  is  with 
respect  to  humans  and  robots  performing  a  task  together,  there  are 
cases  where  the  robot  may  have  to  interact  with  bystanders  or  with 
people  who  are  not  expecting  to  work  with  a  robot.  Examples  include 
the  urban  search  and  rescue  robot  that  comes  across  a  human  to  be 
rescued,  a  military  robot  in  an  urban  environment  that  must  interact 
with  civilians,  and  a  health  assistant  robot  that  must  help  a  patient 
and  interact  with  visitors.  The  role  of  the  robot  with  respect  to  humans 
must  be  taken  into  account.  The  role  of  the  human  will  be  discussed  in 
more  detail  in  Section  5. 

4.4  Adaptation,  Learning,  and  Training 

Although  robot  adaptation  and  learning  have  been  addressed  by  many 
researchers,  training  of  humans  appears  to  have  received  comparatively 
little  attention  in  the  HRI  literature,  even  though  this  area  is  very 
important.  One  reason  for  this  apparent  trend  is  that  an  often  unstated 
goal  of  HRI  is  to  produce  systems  that  do  not  require  significant  train¬ 
ing.  This  may  be  because  many  robot  systems  are  designed  to  be  used 
in  very  specific  domains  for  brief  periods  of  times  [271,  292].  Moreover, 
robot  learning  and  adaptation  are  often  treated  as  useful  in  behavior 
design  and  in  task-specific  learning,  though  adaptation  is  certainly  a 
key  element  of  long-term  interactions  between  humans  and  robots  [104] . 

On  one  hand,  it  is  important  to  minimize  the  amount  of  human 
training  and  adaptation  required  to  interact  with  robots  that  are  used 
in  therapeutic  or  educational  roles  for  children,  autistic  individuals,  or 
mentally  challenged  individuals.  On  the  other  hand,  it  is  important 
that  HRI  include  proper  training  for  problems  that  include,  for  exam¬ 
ple,  handling  hazardous  materials;  similarly  the  very  nature  of  using 
robots  in  therapeutic  and  educational  roles  requires  that  humans  should 
directly  adapt  and  learn  from  the  interaction  [148].  In  this  section,  we 
discuss  not  only  HRI  domains  that  require  minimum  operator  training, 
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but  also  domains  that  require  careful  training.  We  also  discuss  efforts 
aimed  to  train  HRI  scientists  and  designers,  and  then  conclude  with  a 
discussion  of  how  the  concept  of  training  can  be  used  to  help  robots 
evolve  new  skills  in  new  application  domains. 

Minimizing  Operator  Training.  Minimizing  training  appears  to 
be  an  implicit  goal  for  “edutainment”  robots,  which  include  robots 
designed  for  use  in  classrooms  and  museums,  for  personal  entertain¬ 
ment,  and  for  home  use.  These  robots  are  typically  designed  to  be 
manageable  by  a  wide  variety  of  humans,  and  training  can  range  from 
instruction  manuals,  instruction  from  a  researcher,  or  instructions  from 
the  robot  itself  [210,  275].  One  relevant  study  explored  how  ROOMBA 
robots  are  used  in  practice  without  attempting  to  make  operators  use 
the  robots  in  a  specific  way  [99].  Such  studies  are  important  because 
they  can  be  used  to  create  training  materials  that  guide  expectations 
and  alert  humans  to  possible  dangers.  Other  such  studies  include  those 
that  explore  how  children  use  education  robots  in  classroom  settings 
[148],  investigate  how  disabled  children  interact  with  robots  in  social 
settings  [23],  support  humans  in  the  house  [302],  and  identify  interac¬ 
tion  patterns  with  museum  guide  robots  [210]. 

Complementing  such  studies  are  efforts  to  use  archetype  patterns  of 
behavior  and  well-known  metaphors  that  trigger  correct  mental  mod¬ 
els  of  robot  operation.  Examples  include  the  often  stated  hypotheses 
that  people  with  “gaming  experience”  will  be  able  to  interact  better 
(in  some  sense)  with  mobile  robots  than  those  with  limited  experiences 
in  games  [241],  We  are  not  aware  of  any  studies  that  directly  support 
this  hypothesis,  but  if  it  is  true  then  it  would  seem  to  suggest  that 
people  with  experiences  in  video-conferencing,  instant-messaging,  and 
other  computer-mediated  forms  of  communication  might  more  natu¬ 
rally  interact  with  robots.  Whether  this  hypothesis  is  true  is  a  matter 
of  future  work,  but  it  is  almost  certainly  true  that  such  experiences  help 
people  form  mental  models  that  influence  interactions  [238].  Designers 
are  seeking  (a)  to  identify  interaction  modes  that  invoke  commonly  held 
mental  models  [66]  such  as  those  invoked  by  anthropomorphic  robots 
[156]  or  (b)  to  exploit  fundamental  cognitive,  social,  and  emotional 
processes  [32] .  One  possible  caution  for  these  efforts  is  that  robots  may 
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reach  an  “uncanny  valley”  where  expectations  evoked  by  the  robot  fall 
short  of  actual  behavior  producing  an  interaction  that  can  feel  strangely 
uncomfortable  to  humans  [71,  194].  However,  this  uncanny  valley  the¬ 
ory  is  unproven  although  researchers  are  now  trying  to  experimentally 
verify  its  existence  [179]. 

Efforts  to  Train  Humans.  In  contrast  to  the  goal  of  minimizing 
training  in  edutainment  robots,  some  application  domains  involving 
remote  robots  require  careful  training  because  operator  workload  or 
risk  is  so  high.  Important  examples  of  such  training  are  found  in  mil¬ 
itary  and  police  applications,  space  applications,  and  in  search  and 
rescue  applications.  Training  for  military  and  police  applications  is 
typified  by  “bomb  squad”  robots,  training  for  space  applications  is 
typified  by  telemanipulation  tasks  [234],  and  training  for  military  and 
civilian  search  and  rescue  is  typified  by  reconnaissance  using  small, 
“human-packable  robots”  [85] .  In  both  the  military  and  search  applica¬ 
tion  domains,  training  efforts  exist  for  both  air  and  ground  robots,  and 
these  efforts  tend  to  emphasize  the  use  of  mobile  robots  in  a  mission 
context  [87].  Training  efforts  include  instructions  on  using  the  inter¬ 
face,  interpreting  video,  controlling  the  robot,  coordinating  with  other 
members  of  the  team,  and  staying  safe  while  operating  the  robot  in 
a  hostile  environment.  Such  training  is  often  given  to  people  who  are 
already  experts  in  their  fields  (such  as  in  search  and  rescue) ,  but  is  also 
given  to  people  who  may  be  relatively  inexperienced.  In  the  military, 
police  and  space  domains,  training  programs  may  be  complemented  by 
selection  criteria  to  help  determine  which  indviduals  are  likely  to  be 
better  (in  some  sense)  at  managing  a  robot  [79].  Selection  appears  to 
have  received  more  attention  in  air  robots  than  ground  robots. 

By  contrast  to  interactions  with  remote  robots,  many  applica¬ 
tions  involving  proximate  robots  are  designed  to  produce  learning  or 
behavioral  responses  with  humans.  Therapeutic  and  social  robots  are 
designed  to  change,  educate,  or  train  people,  especially  in  long-term 
interactions  [148,  245,  312],  People  also  adapt  to  service  robots  over  the 
long-term  and  over  a  wide  range  of  tasks  [115],  and  there  is  growing 
evidence  that  many  long-term  interactions  require  mutual  adaptation 
including  with  human  bystanders  [75,  131,  117].  Importantly,  culture 
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appears  to  influence  both  long-term  and  short-term  adaptation,  at  least 
as  far  as  accepting  interactions  with  a  robot  [18,  94,  127,  150,  270,  306]. 

Training  Designers.  Importantly,  an  often  overlooked  area  is  the 
training  of  HRI  researchers  and  designers  in  the  procedures  and 
practices  of  those  whom  they  seek  to  help.  Important  examples  of 
training  researchers  include  Murphy’s  workshops  on  search  and  res¬ 
cue  robotics  [198],  tutorials  and  workshops  on  methodologies  for 
understanding  a  work-practice  domain  and  field  studies  [58],  tuto¬ 
rials  for  young  researchers  on  search  and  rescue  [239,  240],  and 
tutorials  and  workshops  on  metrics  or  experiment  design  for  robot 
applications  [294], 

Training  Robots.  It  is  tempting  to  restrict  training  to  the  education 
of  the  human  side  of  HRI,  but  this  would  be  a  mistake  given  current 
HRI  research.  In  HRI,  robots  are  also  learning,  both  offline  as  part  of 
the  design  process  [37,  207]  and  online  as  part  of  interaction,  especially 
long-term  interaction  [89,  251].  Such  learning  includes  improving  per¬ 
ceptual  capabilities  through  efficient  communication  between  humans 
and  robots  [33,  89,  251,  330],  improving  reasoning  and  planning  capa¬ 
bilities  through  interaction  [295,  34],  and  improving  autonomous  capa¬ 
bilities  [258].  Approaches  to  robot  learning  include  teaching  or  pro¬ 
gramming  by  demonstration  [13,  55,  76,  77,  146,  218],  task  learning 
[34,  251,  207],  and  skill  learning  including  social,  cognitive,  and  loco¬ 
motion  skills  [11,  22,  202,  233,  295].  Some  researchers  are  exploring 
biologically  inspired  learning  models,  including  how  teaching  among 
humans  or  social  animals  can  be  used  to  train  a  robot  [11,  257]; 
others  are  exploring  how  learning  can  become  more  efficient  if  it 
leverages  information  about  how  the  human  brain  learns  in  very  few 
trials  [100]. 

Interestingly,  it  can  be  argued  that  providing  support  for  effi¬ 
cient  programming  or  knowledge  management  systems  is  an  impor¬ 
tant  aspect  of  training  robots  in  HRI  [120,  327].  Additionally,  it  can 
be  argued  that  sensitizing  a  robot  to  issues  of  culture  and  etiquette 
allows  them  to  adapt  to  slowly  changing  human  norms  of  behavior 
[141,  215,  252], 
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4.5  Task-Shaping 

Robotic  technology  is  introduced  to  a  domain  either  to  allow  a  human 
to  do  a  task  that  they  could  not  do  before,  or  to  make  the  task 
easier  or  more  pleasant  for  the  human.  Implicit  in  this  assertion  is 
the  fact  that  introducing  technology  fundamentally  changes  the  way 
that  humans  do  the  task.  Task-shaping  is  a  term  that  emphasizes 
the  importance  of  considering  how  the  task  should  be  done  and  will 
be  done  when  new  technology  is  introduced.  Compared  to  the  other 
ways  that  a  designer  can  shape  HRI,  there  is  little  written  about 
task-shaping. 

There  are  formal  processes  for  understanding  how  the  task  should 
be  done  and  is  currently  done.  These  processes  include  goal-directed 
task  analyses,  cognitive  work  analyses,  and  ethnographic  studies  [58, 
88,  301].  Although  frequently  used  to  specify  how  a  task  is  done  and 
how  it  should  be  done,  it  is  imperative  to  consider  how  the  task  will  be 
done,  including  unintended  consequences  of  design  [14,  212], 

One  reason  that  little  is  written  about  task-shaping  is  because 
designers  are  implicitly  trying  to  create  technology  and  interactions 
that  accomplish  some  task  or  function.  Indeed,  Woods  has  persuasively 
argued  that  designing  a  system  is  equivalent  to  making  a  hypothesis 
about  how  the  artifact  will  positively  shape  the  experience  [319].  Nev¬ 
ertheless,  it  is  important  to  consider  how  the  task  might  be  modified  to 
better  support  interaction.  Examples  of  explicit  task-shaping  include 
designing  space  or  underwater  equipment  and  tools  so  that  handles 
and  connectors  can  be  manipulated  by  a  robotic  arm,  “pre-cleaning”  a 
room  so  that  a  robot  vacuum  can  accomplish  its  task  most  efficiently 
[99] ,  and  performing  pre- inspection  tasks  used  to  form  maps  and  plans 
that  can  be  executed  by  a  robot. 


4.6  Finding  a  Unifying  Theme 

There  are  multiple  ways  to  frame  HRI  as  a  field.  One  approach  is 
to  treat  HRI  as  a  resurgent  emphasis  and  extension  of  previous  work 
in  human  factors,  teleoperation,  and  supervisory  control.  Another 
approach  to  framing  HRI  is  to  view  it  as  a  new  field  that  includes  a  con- 
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vergence  of  previous  work  with  new  research  problems  caused  by  some 
new  capability  that  fundamentally  changes  the  problem.  We  assert  that 
robot  autonomy  has  reached  the  point  where  mixed-initiative  interac¬ 
tion  and  semi-autonomous  control  have  fundamentally  changed  the  field 
from  previous  research  on  related  problems.  Thus,  we  treat  HRI  as  a 
new  field  that  faces  opportunities  and  problems  which  are  not  simple 
extensions  of  previous  work.  We  acknowledge,  however,  that  it  is  pos¬ 
sible  to  make  persuasive  arguments  that  HRI  is  simply  a  refocusing  of 
previous  efforts  rather  than  a  new  field. 

One  way  to  unify  the  scope  of  current  HRI  research  is  to  con¬ 
dense  the  five  dimensions  of  designer  influence  into  a  single  concept 
as  exemplified  in  our  proposed  scale  of  interaction,  Figure  4.2,  with  the 
caveat  that  this  single  concept  cannot  capture  every  nuance  and  possi¬ 
ble  design  of  every  HRI  problem.  The  concept  of  dynamic  interaction 
seems  to  capture  the  current  research  direction  of  many  HRI  efforts. 

Dynamic  interaction  includes  time-  and  task-varying  changes  in 
autonomy,  information  exchange,  team  organization  and  authority, 
and  training.  It  applies  to  both  remote  and  proximate  interactions, 
including  social  and  physical  interactions.  By  including  variable  auton¬ 
omy  assignments,  the  concept  of  dynamic  interaction  subsumes  adap¬ 
tive  and  dynamic  autonomy  as  a  special  case  [80,  96,  97,  111,  144, 
165,  259].  By  including  information  exchange,  dynamic  interaction 
includes  adaptive  and  adaptable  interfaces  [144,  153,  221],  By  includ¬ 
ing  team  organization  and  authority,  mixed  initiative  interaction  [43, 
98,  162,  226]  is  addressed.  By  including  training,  interactive  learning 
is  included. 

More  importantly,  the  concept  of  dynamic  interaction  places  the 
emphasis  on  shaping  the  types  of  interactions  that  can  and  will 
emerge  as  humans  and  robots  interact.  The  scope  of  HRI  research 
and  design,  therefore,  includes  all  efforts  at  evaluating  systems  and 
interaction  paradigms,  designing  autonomy  algorithms  in  the  con¬ 
text  of  HRI,  designing  interfaces  and  information  exchange  protocols, 
defining  and  switching  roles,  and  influencing  learning  and  training. 
This  emphasis  on  dynamic  interaction  differs  sharply  from  the  histor¬ 
ically  static  interactions  of  pure  teleoperation  and  pure  supervisory 
control. 
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Note  that  some  current  research  efforts  and  methods  do  not  natu¬ 
rally  fit  into  the  dynamic  interaction  framework.  These  include  several 
aspects  of  task  shaping,  including  ethnographic  studies,  goal-directed 
task  analyses,  and  some  cognitive  science-based  work.  However,  under¬ 
standing  existing  processes  and  potential  use  patterns  helps  researchers 
better  understand  the  fluid  interaction  patterns  that  are  likely  to  exist 
in  practice  so  that  they  can  design  interactions  that  support,  improve, 
and  extend  these  interaction  patterns. 


5 


Problem  Domains  in  HRI 


We  have  already  mentioned  several  of  the  problem  domains  and  appli¬ 
cation  areas  in  modern  HRI.  In  this  section,  we  elaborate  on  many  of 
these  problem  domains  to  present  a  survey  of  the  kinds  of  problems 
encountered  in  HRI.  Importantly,  many  of  these  problems  have  broad 
social  impact;  thus,  much  work  in  HRI  uses  science  and  engineering  to 
respond  to  needs  in  society. 

Scholtz  provided  a  taxonomy  of  roles  that  robots  can  assume  in 
HRI  [262]: 

•  Supervisor, 

•  Operator, 

•  Mechanic, 

•  Peer,  and 

•  Bystander. 

To  this  list,  we  add  the  following: 

•  Mentor:  the  robot  is  in  a  teaching  or  leadership  role  for  the 
human,  and 
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•  Information  Consumer:  the  human  does  not  control  the 
robot,  but  the  human  uses  information  coming  from  the 
robot  in,  for  example,  a  reconnaissance  task. 


Similar  taxonomies  are  certainly  possible,  but  identifying  how  people 
perceive  a  robot’s  role  has  important  ramifications  for  how  they  inter¬ 
act  with  the  robot  [74],  Using  Scholtz’s  taxonomy  provides  insight  into 
the  current  and  future  interactions  in  these  applications.  Table  5.1  clas¬ 
sifies  the  most  frequent  types  of  interactions  for  the  application  areas 
discussed  in  the  remainder  of  this  section.  For  many  of  these  areas,  cur¬ 
rent  research  patterns  exhibit  a  trend  away  from  remote  interactions 


Table  5.1  Examples  of  roles  and  proximity  patterns  that  arise  in  several  application  areas. 


Application  area 

Remote/ 

Proximate 

Role 

Example 

Search  and  rescue 

Remote 

Human  is  supervisor  or 
operator 

Remotely  operated  search 
robots 

Proximate 

Human  and  robot  are 

peers 

Robot  supports  unstable 
structures 

Assistive  robotics 

Proximate 

Human  and  robot  are 
peers,  or  robot  is  tool 

Assistance  for  the  blind, 
and  therapy  for  the 
elderly 

Proximate 

Robot  is  mentor 

Social  interaction  for 
autistic  children 

Military  and  police 

Remote 

Human  is  supervisor 

Reconnaissance , 
de-mining 

Remote  or 
Proximate 

Human  and  robot  are 

peers 

Patrol  support 

Remote 

Human  is  information 

consumer 

Commander  using 
reconnaissance 

information 

Edutainment 

Proximate 

Robot  is  mentor 

Robot  is  mentor 

Robot  is  peer 

Robotic  classroom 

assistant 

Robotic  museum  tour 
guide 

Social  companion 

Space 

Remote 

Human  is  supervisor  or 
operator 

Remote  science  and 
exploration 

Proximate 

Human  and  robot  are 

peers 

Robotic  astronaut 

assistant 

Home  and  industry 

Proximate 

Human  and  robot  are 

peers 

Robotic  companion 

Proximate 

Human  is  supervisor 

Robotic  vacuum 

Remote 

Human  is  supervisor 

Robot  construction 
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toward  proximate  interactions,  and  away  from  operator  roles  toward 
peer  or  mentor  roles. 

5.1  Search  and  Rescue 

The  highest  profile  HRI  research  area  in  the  United  States  is  a  strong 
example  of  work  with  important  social  impact.  This  area  is  urban 
search  and  rescue  (USAR),  and  is  exemplified  by  the  use  of  robots 
in  rescue  and  recovery  efforts  after  the  collapse  of  the  World  Trade 
Center  buildings  [51].  Lead  by  pioneering  government  and  academic 
efforts  [26,  200],  USAR  has  grown  into  one  of  the  most  important  areas 
of  HRI. 

Because  of  its  importance,  USAR  has  become  an  HRI  challenge 
problem.  As  such,  there  have  been  efforts  to  provide  standardized 
USAR  test  areas  and  performance  measures,  and  to  standardize 
robot-assisted  USAR  efforts  [138,  199,  263].  There  are  regular  USAR 
competitions  at  robot-related  conferences  [159,  199,  325].  A  wide  vari¬ 
ety  of  interface  concepts,  autonomy  designs,  sensor-processing  algo¬ 
rithms,  robot  morphologies,  field  studies,  and  human  factors  analyses 
and  experiments  have  been  created  in  the  name  of  robot-assisted  USAR 
(see,  for  example,  [197,  198,  211,  264,  318]).  Recently,  these  efforts  are 
being  extended  from  ground  robots  to  include  aerial  robots  used  in 
natural  disaster  and  wilderness  search  [61,  278]. 

5.2  Assistive  and  Educational  Robotics 

In  the  spirit  of  socially  relevant  research,  robots  are  being  developed 
to  serve  in  assistive  and  educational  capacities.  Assistive  robotics  is 
perhaps  one  of  the  highest  profile  areas  of  HRI  in  the  world.  This 
application  domain  often  places  the  robot  in  a  peer-like  or  mentoring 
role  with  the  human  in  practice,  even  though  the  intention  of  the  robot 
is  designed  to  provide  service  to  the  human.  For  example,  robots  being 
developed  to  assist  the  visually  challenged  work  in  close  proximity  to 
humans  and  must  merit  an  appropriate  level  of  trust.  The  goal  of  work 
in  this  area  is  to  increase  the  set  of  tasks  that  a  visually  impaired  per¬ 
son  can  independently  perform.  These  tasks  include  providing  naviga- 
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tion  assistance  in  unstructured  domains  [167,  170,  274],  and  providing 
information  about  locations  (and  prices)  in  grocery  stores  and  trans¬ 
portation  facilities  [168]. 

Another  important  area  of  research,  especially  in  countries  with  a 
burgeoning  population  of  the  elderly,  is  providing  support  for  those 
who  have  age-related  challenges  [192,  231,  232,  252,  304],  Researchers 
in  this  area  focus  on  both  physical  needs  such  as  mobility  assistance 
[324],  emotional  welfare  [252,  303],  and  cognitive  assistance  [151].  Many 
working  in  this  area  are  fully  aware  of  the  ethical  considerations  that 
arise  by  delegating  a  companionship  role  to  a  robot  and  the  trust  issues 
that  arise  by  having  an  artificial  agent  working  with  someone  with 
mental  challenges  such  a  senility  [286].  Having  the  ability  to  remotely 
manage  a  personal  assistant  robot  over  the  internet  may  be  a  step  in 
the  direction  of  addressing  these  ethical  concerns,  though  substantial 
technological  and  privacy  limitations  exist  for  such  interaction  [166]. 

For  some  people  with  physical  and  mental  challenges,  robots  may 
provide  an  opportunity  for  interaction  and  therapy.  Such  work  is  being 
explored  with  autistic  children  [23,  247] .  Many  of  these  children  respond 
weakly  or  not  at  all  to  social  cues,  but  respond  well  to  mechanical 
devices.  Robots  provide  a  possible  therapeutic  role  for  using  a  mechan¬ 
ical  device  to  improve  social  interactions  [312],  Robots  are  also  being 
considered  for  other  domains  where  children  are  benefited,  such  as  those 
who  have  experienced  trauma.  Importantly,  the  social  dimension  of  HRI 
is  considered  necessary  not  only  in  assistive  roles  but  also  in  many  areas 
of  proximate  interaction  [31,  32,  72,  90,  106,  148].  Indeed,  the  area  of 
social  robotics  is  so  large  that  it  is  already  time  to  update  the  highly 
cited  2003  survey  [94], 

For  some  people  with  physical  challenges,  the  embodiment  of  a  robot 
provides  unique  opportunities  not  available  in  other  forms  of  technol¬ 
ogy.  For  example,  researchers  are  working  on  designing  robots  that  pro¬ 
vide  support  for  physical  therapy.  Efforts  include  providing  prescribed 
force  and  movement  trajectories  to  help  rebuild  flexibility  and  strength 
[178].  Other  work  includes  detecting  motivational  state  and  adjusting 
therapy  to  maximize  benefits  [236].  Intelligent  wheelchairs  are  a  type 
of  robot  that  uses  external  sensors  to  support  path  planning,  collision 
avoidance,  etc.  for  a  person  that  requires  a  wheelchair  [134,  324]. 


5.3  Entertainment  237 


Physical  interaction  with  robots  is  not  limited  to  providing  assis¬ 
tance  to  those  with  some  form  of  disability.  Many  robot  architectures 
have  been  designed  to  assist  humans  in  industrial  settings  [228].  Indeed, 
developing  telerobots  for  use  in  hazardous  materials  handling  has  been 
ongoing  for  several  decades  [139].  However,  cooperative  systems  are 
not  limited  to  large-scale  industrial  settings,  but  may  be  most  useful  in 
small-scale  tasks  “such  as  nricroassembly  and  microsurgery”  [180].  The 
technologies  developed  for  such  assistive  support  appear  to  apply  not 
only  to  proximate  physical  interaction,  but  also  to  the  general  telenra- 
nipulation  problem  [224]. 

In  the  spirit  of  providing  assistance  to  those  without  a  disability, 
some  researchers  are  also  exploring  how  robots  can  be  used  to  promote 
education  for  typical  children,  both  in  the  home  and  in  schools  [62,  140]. 
The  personal  service  robot  is  an  extension  to  the  theme  of  providing 
service  to  the  general  population.  Indeed,  South  Korea  has  ambitious 
goals  about  the  advancement  of  such  robots  [219]. 

5.3  Entertainment 

Although  there  have  been  many  examples  of  entertainment  robotics, 
from  an  HRI  perspective,  not  much  has  been  published  in  the  liter¬ 
ature.  Early  entertainment  robotics  centered  on  animatronics,  where 
the  robot  generally  plays  prerecorded  sounds  that  are  synchronized 
with  the  robots  motion.  These  types  of  robots  can  often  be  found  in 
old  movies  and  theme  parks;  however,  the  interaction  is  mostly  in  one 
direction,  that  of  the  robot  presenting  information,  although  the  robot’s 
performance  may  be  triggered  by  the  presence  of  the  human.  However, 
the  2005  AICHI  Expo  demonstrated  several  robots  designed  to  enter¬ 
tain,  including  the  use  of  robots  as  actors  and  dance  partners  [6] ;  similar 
work  on  the  relationship  between  acting,  drama,  and  artificial  agents  is 
presented  in  [41].  Here  again,  the  role  of  the  human  is  as  an  observer, 
and  the  interaction  is  minimal  and  more  implicit  [41]. 

The  Valerie  and  Tank  robots  at  Carnegie  Mellon  University  were 
designed  as  robot  receptionists  as  a  joint  project  between  CMU’s 
Robotics  Institute  and  the  School  of  Drama  [106].  The  robots  have  a 
rich  back  story  and  strive  to  increase  users’  interaction  by  encouraging 
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the  user  to  ask  about  the  robots’  lives  outside  of  their  role  as  recep¬ 
tionists.  Although  robots  that  act  as  tour  guides  might  be  considered 
a  form  of  assistive  robotics,  we  classify  such  robots  under  the  category 
of  entertainment  because  their  primary  role  is  to  engage  participants 
[120,  210]. 

Although  not  strictly  entertainment,  the  Insect  Telepresence  project 
[8]  placed  Madagascar  Hissing  Cockroaches  in  a  terrarium,  and  allowed 
users,  through  mediated  telepresence,  to  drive  a  miniature  camera 
within  the  terrarium,  at  eye  level  to  the  cockroaches.  The  image  from 
the  camera  was  projected  much  larger  than  life  on  a  wall.  Of  interest 
was  the  mediated  teleoperation,  which  limited  the  accelerations  of  the 
camera  motions  to  below  that  which  would  cause  the  cockroaches  to 
react  to  the  camera,  offering  the  humans  a  look  into  the  cockroach’s 
world. 

Other  HRI-related  research  in  the  use  of  robots  for  entertainment 
include  robotic  story  tellers  [191],  robotic  dance  partners  [163],  robotic 
plants  that  give  users  information  such  as  incoming  email  [137],  and 
robotic  pets  [94,  272], 

5.4  Military  and  Police 

Many  have  called  for  robots  to  be  used  in  tasks  that  are  “dull,  dirty, 
or  dangerous.”  Military  and  police  applications  often  simultaneously 
satisfy  all  three  criteria  for  robot  use.  Applications  include  gathering 
information  to  support  a  dangerous  task  such  as  a  SWAT  team  take¬ 
down,  or  using  remote  vehicles  in  combat  to  minimize  risk  exposure 
to  soldiers.  Current  work  emphasizes  robots  as  servants  to  soldiers  and 
officers  [87],  typically  in  remote  operations,  but  efforts  to  have  robots 
work  in  a  peer-like  role  are  underway  [44,  155]. 

A  typical  use  of  robots  in  both  military  and  police  applications  is 
in  bomb  disposal  (called  “improvised  explosive  devices”  in  many  mil¬ 
itary  situations)  [263] .  Remotely  controlled  robots  are  frequently  used 
to  approach  and  evaluate  suspicious  packages  [311].  Controlling  these 
robots  is  demanding  on  the  operators,  especially  since  many  current 
interfaces  and  autonomy  levels  require  operators  to  integrate  informa¬ 
tion  from  multiple  sources  of  data  such  as  multiple  cameras.  In  addition 
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to  controlling  mobile  platforms,  telenranipulation  is  an  important  part 
of  such  work.  Robot  arms  are  often  mounted  on  the  platforms  providing 
operators  some  ability  to  manipulate  the  object.  Because  of  the  lim¬ 
ited  situation  awareness  for  controlling  these  arms,  some  researchers  are 
exploring  sensor  technologies  and  information  presentation  techniques 
that  improve  awareness  in  telenranipulation  [242]. 

5.5  Space  Exploration 

Robots  have  long  been  part  of  space  exploration.  In  fact,  according  to 
some  definitions,  a  satellite  can  be  considered  a  type  of  robot,  albeit 
one  that  has  a  high  degree  of  autonomy  and  typically  requires  minimal 
intervention. 

As  has  been  previously  noted,  remarkable  success  in  space  robotics 
include  the  Soviet  Lunokhods  which  were  used  to  explore  the  surface 
of  the  moon  [95]  followed  by  more  recent  NASA  success  in  exploring 
the  surface  of  Mars  [174,  317].  It  is  anticipated  that  the  robots  will 
continue  to  have  a  strong  role  in  envisioned  and  future  explorations  of 
the  lunar  and  Martian  surfaces,  in  construction  tasks  on  these  surfaces 
as  well  as  in  the  international  space  station,  and  in  remote  science  and 
maintenance  tasks  [84,  129,  177,  225]. 

Many  precursor  and  early  human  missions  will  depend  heavily  on 
remotely  managed  robots,  but  will  also  likely  include  extravehicular 
activities.  To  prepare  for  these  missions,  NASA  and  other  international 
space  agencies  engage  in  frequent  field  work  designed  to  evaluate  both 
the  robotic  and  HRI  technologies.  Examples  of  this  include  develop¬ 
ment  of  the  Dante  robots  [16,  17],  a  series  of  Extra- Vehicular  Activ¬ 
ities  (EVA)-based  field  tests  [47,  114,  296],  and  field  tests  involving 
substantial  communications  delays  and  remote  science  in  harsh  envi¬ 
ronments  [283,  310].  Research  that  extends  these  efforts  includes  the 
aforementioned  Robonaut  development  and  work  on  developing  intel¬ 
ligent  rovers  and  robot  “mules”  to  pack  astronaut  equipment  on  long 
duration  EVAs. 

Complementing  the  focus  on  remote  and  EVA-based  interaction 
are  efforts  to  support  astronauts  at  the  international  space  station 
and  on  long  duration  space  missions.  These  efforts  focus  on  astronaut 
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assistants  and  small  satellites  [54,  81].  Many  of  the  NASA  programs 
include  strong  planning  components  and  seek  to  integrate  software, 
robotic,  and  human  agents  [30].  Other  work  calls  for  standardization 
of  robot  parts,  procedures,  communications,  and  interfaces  [91,  92], 

5.6  UAV  Reconnaissance  and  UUV  Applications 

Unmanned  Air  Vehicles  are  rapidly  attracting  attention  as  an  applica¬ 
tion  area  for  HRI  and  for  aviation  technology.  UAVs  have  been  around 
for  a  long  time  and  have  a  large  body  of  related  literature,  but  the  label 
for  such  vehicles  varies  over  time.  Among  other  labels,  UAV  work  has 
been  published  under  names  like  remotely  piloted  vehicles,  uninhabited 
air  vehicles,  autonomous  micro  air  vehicles,  and  autonomous  aerial  sys¬ 
tems.  The  key  defining  aspect  of  UAV  interaction  is  the  fact  that  the 
vehicles  are,  by  definition,  remotely  operated,  move  in  three  dimen¬ 
sions,  and  typically  have  six  degrees  of  freedom.  Interestingly,  remote 
interactions  and  a  high  degree  of  freedom  are  also  defining  attributes 
for  problems  that  use  Unmanned  Underwater  Vehicles  (UUVs),  often 
referred  to  as  autonomous  underwater  vehicles.  Because  of  this  simi¬ 
larity,  we  unite  our  discussion  of  these  two  important  areas  of  HRI. 
This  unification  may  be  particularly  relevant  because  the  literature 
from  these  two  communities  rarely  overlaps  even  though  the  interac¬ 
tion  dynamics  have  marked  similarities. 

In  addition  to  the  previously  mentioned  reconnaissance-based  mili¬ 
tary  and  search-and-rescue  uses  of  UAVs,  there  are  a  number  of  other 
envisioned  uses  of  these  vehicles.  These  include  atmospheric  science 
[107],  landing  site  surveillance  of  a  Mars  rover  [19],  border  patrol  [103], 
pollution  monitoring  [113],  forest  fire  monitoring  [49],  infrastructure 
inspection  [113],  and  munitions-based  military  applications  [70].  UUVs 
have  similarly  broad  areas  of  use.  They  include  undersea  science,  trea¬ 
sure  hunting,  undersea  and  surf-zone  de-mining,  and  underwater  con¬ 
struction  and  maintenance  [329].  For  remote  operations,  there  are  many 
interface  designs  that  apply  to  both  UAV  and  UUV  operations.  These 
include  integrating  multiple  perspectives,  developing  a  sense  of  telep¬ 
resence,  synchronizing  frames  of  reference  through  tethers,  and  building 
mosaics  [101,  230]. 
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5.7  Other  Applications 

There  are  a  host  of  other  application  domains  where  robots  are  used. 
These  domains  include  home  use,  manufacturing,  inventory  manage¬ 
ment,  mining,  and  precision  agriculture.  Importantly,  some  of  these 
application  domains  include  strong  HRI  research,  but  others  attempt 
the  creation  of  “fully  autonomous”  robots  that  do  not  require  human 
interaction  and,  ironically,  are  not  “autonomous  enough”  to  allow  inter¬ 
action  with  humans.  Although  it  is  possible  that  some  of  these  domains 
might  allow  “fire-up  and  forget”  robots,  it  is  likely  that  the  usefulness 
and  safety  of  robots  in  many  of  these  domains  will  increase  if  HRI 
considerations  are  included  in  their  design. 


6 


Solution  Themes,  Scientific  Approaches, 
and  Challenge  Problems 


One  measure  of  the  maturity  of  a  research  field  is  the  emergence  of 
a  series  of  accepted  practices  and  challenge  problems  that  focus  the 
attention  of  the  field.  Equally  important  is  the  identification  of  solu¬ 
tion  themes  that  cross  applications.  In  this  section,  we  survey  several 
practices,  challenge  problems,  and  solution  themes. 

6.1  Accepted  Practices 

There  are  a  number  of  accepted  practices  that  are  emerging  in  HRI. 
A  key  practice  is  to  include  experts  from  multiple  disciplines  on  research 
efforts.  These  disciplines  frequently  include  robotics,  electrical  and 
mechanical  engineering,  computer  science,  human-computer  interac¬ 
tion,  cognitive  science,  and  human  factors  engineering.  Other  relevant 
disciplines  include  design,  organizational  behavior,  and  the  social  sci¬ 
ences.  Importantly,  some  conferences  encourage  multidisciplinary  sub¬ 
missions  are  working  to  establish  the  practice  of  having  all  papers  ref¬ 
ereed  by  reviewers  representing  different  disciplines  [265]. 

A  second  emerging  practice  is  to  create  real  systems  (robot  auton¬ 
omy,  interaction  modes,  and  etc.)  and  then  evaluate  these  systems 
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using  experiments  with  human  subjects.  Proof-of-concept  technologies, 
although  important,  are  less  valuable  than  they  would  be  if  they  were 
supported  by  careful  experiments  that  identify  key  attributes  of  the 
design  or  principles  that  span  applications.  Identification  of  descrip¬ 
tive  interaction  phenomena  is  interesting,  but  elaboration  on  the  psy¬ 
chological  principles  underlying  these  phenomena  with  an  eye  toward 
harnessing  these  principles  in  design  is  more  useful.  Thus,  engineering, 
evaluation,  and  modeling  are  key  aspects  of  HRI. 

A  third  emerging  practice  is  conducting  experiments  that  include 
a  careful  blending  of  results  from  simulated  and  physical  robots.  On 
the  one  hand,  because  of  cost  and  reliability  issues,  it  is  often  difficult 
to  conduct  carefully  controlled  experiments  with  physical  robots.  On 
the  other  hand,  it  is  often  difficult  to  replicate  simulation-only  results 
with  physical  robots  because  the  physical  world  presents  challenges 
and  details  that  are  not  present  in  many  simulations.  It  is  common 
to  “embody”  at  least  one  portion  of  the  interaction,  be  it  a  physi¬ 
cal  robot,  some  physical  sensor,  or  real-world  speech.  Some  research 
includes  work  using  carefully  controlled  simulation  environments  and 
replication  of  selected  results  with  physical  robots.  Others  use  wizard 
of  Oz  studies.  Others  form  communities  where  roboticists  design  tech¬ 
nologies  and  where  other  human  factors  researchers  collect  and  analyze 
results  from  tests;  this  is  the  operating  structure  of  part  of  the  robot- 
assisted  urban  search  and  rescue  (USAR)  community  [325].  Interest¬ 
ingly,  at  least  one  research  group  is  exploring  how  a  simulated  user  can 
help  support  the  design  of  human-robot  interfaces  [244] . 

A  fourth  area  of  emerging  effort  is  establishing  standards  and  com¬ 
mon  metrics.  The  most  complete  survey  of  metrics  is  in  [280],  but 
much  work  on  metrics  exists  in  the  literature  including  the  proceedings 
of  the  annual  PERMIS  workshops.  Standardization  efforts  have  been 
strongest  in  the  USAR  domain  [138,  199],  but  are  also  present  in  space 
applications  and  UAVs  [91,  92]. 

A  fifth  emerging  practice  is  the  use  of  longitudinal  studies.  Such 
studies,  which  can  last  from  several  weeks  to  several  months,  require  a 
considerable  investment  by  researchers,  both  in  terms  of  hours  and 
financial  resources.  One  reason  that  long-term  studies  are  a  rela¬ 
tively  recent  practice  is  that  many  robots  were  not  reliable  enough 


244  Solution  Themes,  Scientific  Approaches,  and  Challenge  Problems 


to  work  over  the  study  period.  The  availability  of  reliable  per¬ 
sonal  home  robots  and  service  robots  in  public  areas  has  made  such 
studies  possible  [99,  120].  The  European  COGNIRON  project  is  a 
good  example  of  a  commitment  to  long-term  studies  [59].  Long¬ 
term  studies  shift  research  methodologies  from  carefully  controlled 
small-scale  experiments  to  other  methodologies  such  as  surveys  and 
ethnography. 

6.2  Challenge  Problems 

It  is  often  useful  to  identify  a  set  of  challenge  problems  that  focus  the 
efforts  of  a  community.  HRI  has  a  suite  of  challenge  problems,  some 
explicitly  identified  as  such  and  others  implicitly  operating  as  such.  In 
this  section,  we  identify  a  collection  of  problems  that  are  likely  to  shape 
HRI  in  the  near  future.  For  each  problem,  we  discuss  those  aspects  of 
the  problem  that  make  it  particularly  challenging  and  useful. 

USAR  is  the  most  high  profile  of  the  HRI  challenge  problems.  The 
attributes  that  define  this  as  a  challenge  problem  include  the  highly 
unstructured  nature  of  USAR  environments.  This  imposes  strict  chal¬ 
lenges  on  robot  mobility,  communications,  map-building,  and  situation 
awareness. 

Developing  robots  to  be  used  in  military  reconnaissance  and  com¬ 
bat  is  another  high  profile  challenge  area  in  HRI.  Similar  to  USAR, 
environments  tend  to  be  unstructured,  but  perhaps  more  importantly 
operators  may  be  required  to  operate  under  extreme  stress  in  the  pres¬ 
ence  of  an  adversary  that  is  trying  to  prevent  their  success. 

Space  robotics  is  another  area  where  the  environment  is  often 
unstructured,  and  the  environment  is  often  extreme  in  terms  of  tem¬ 
perature,  radiation,  the  vacuum  of  space,  and  the  presence  of  dust. 
Important  characteristics  of  space  robotics  include  the  observation 
that  operators  can  be  highly  trained,  but  communications  may  be 
very  limited  due  to  time  delays,  power  limitations,  and  even  opera¬ 
tor  mobility  (as  in  the  case  of  an  astronaut  interacting  from  within  a 
space  suit). 

Assistive  robotics  is  a  challenge  area,  but  not  because  the  environ¬ 
ment  is  unstructured  per  se.  Rather,  the  key  attributes  of  this  problem 
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are  the  proximity  and  vulnerability  of  the  human  in  the  interaction 
plus  the  potential  for  interactions  that  may  evolve  in  unanticipated 
patterns. 

Humanoid  robotics  is  a  challenge  area,  both  in  terms  of  engineer¬ 
ing  human-like  movements  and  expressions,  and  in  terms  of  the  chal¬ 
lenges  that  arise  when  a  robot  takes  a  human  form.  With  such  a  form, 
social  and  emotional  aspects  of  interaction  become  paramount. 

Natural  language  interaction  is  a  challenge  problem,  not  only 
because  it  requires  sophisticated  speech  recognition  and  language 
understanding,  but  also  because  it  inevitably  includes  issues  of  mixed- 
initiative  interaction,  multi-modal  interaction,  and  cognitive  modeling. 

6.3  Solution  Themes 

HRI  presents  a  number  of  problems  that  cross  application  domains. 
These  problems  include  requirements  on  autonomy,  information  shar¬ 
ing,  and  evaluation.  Emerging  from  these  problems  are  a  set  of  solu¬ 
tion  themes  that  cross  applications  and  that,  when  addressed,  can  be 
leveraged  across  several  problems.  In  this  section,  we  identify  some  of 
these  solution  themes  and  discuss  some  of  the  open  questions  associated 
with  them. 

Dynamic  Autonomy,  Mixed-Initiative  Interaction,  and  Dialog. 

Because  most  interesting  applications  of  human-robot  interaction 
include  rich  information  exchanges  in  dynamic  and  complex  environ¬ 
ments,  it  is  imperative  that  interactions  and  resulting  behaviors  can 
accommodate  complexity. 

Telepresence  and  Information  Fusion  in  Remote  Interaction. 

Although  remote  control  and  teleoperation  are  the  oldest  forms  of 
human-robot  interaction,  the  problem  is  far  from  solved.  In  fact,  with 
advances  in  robot  morphology,  sensor  processing,  and  communications, 
it  is  necessary  to  find  new  ways  to  fuse  information  to  provide  humans 
an  operational  presence  with  the  robot.  Obstacles  to  achieving  this 
include  bandwidth  limitations,  communications  delays  and  drop-outs, 
mismatches  in  frames  of  reference,  communicating  intent  and  trusting 
autonomy,  and  mismatches  between  expectations  and  behaviors. 
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Cognitive  Modeling.  Effective  interactions  between  humans  include 
a  common  ground  created  by  common  experiences  and  cultures.  This 
common  ground  creates  realistic  expectations  and  forms  the  basis  com¬ 
munications.  From  a  robot’s  perspective,  supporting  effective  interac¬ 
tions  also  requires  establishing  and  maintaining  common  ground.  An 
emerging  approach  to  doing  this  is  to  create  cognitive  models  of  human 
reasoning  and  behavior  selection.  The  goal  is  to  create  rich  enough  mod¬ 
els  either  (a)  to  allow  the  robot  to  identify  a  human’s  cognitive  state 
and  adjust  information  exchange  accordingly  or  (b)  to  allow  the  robot’s 
behavior  to  be  generated  by  models  that  are  interpretable  by  a  human. 

Team  Organizations  and  Dynamics.  Many  HRI  researchers  are 
striving  to  develop  systems  that  allow  multiple  robots  and  multiple 
humans  to  interact  with  each  other.  To  accomplish  this,  it  is  necessary 
to  shape  team  interactions  and  dynamics  by  establishing  organizational 
structures,  communications  protocols,  and  support  tools.  Team  organi¬ 
zations  necessarily  subsume  different  and  dynamic  roles,  which  implies 
that  such  efforts  will  need  to  leverage  lessons  from  research  on  mixed 
initiative  and  dialog. 

Interactive  Learning.  Because  the  world  is  complex,  interactions 
between  humans  and  robots  are  also  complex.  This  implies  that  it 
is  impossible  to  anticipate  every  conceivable  problem  and  generate 
scripted  responses,  or  anticipate  every  conceivable  percept  and  gen¬ 
erate  sensor  processing  algorithms.  Interactive  learning  is  the  process 
by  which  a  robot  and  a  human  work  together  to  incrementally  improve 
perceptual  ability,  autonomy,  and  interaction. 
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Although  we  have  framed  HRI  as  a  new  field  in  this  review,  HRI  has 
strong  ties  to  previous  and  ongoing  work  in  telerobotics,  intelligent 
vehicle  systems,  human-computer  interaction,  etc.  In  this  section,  we 
review  many  of  the  stronger  ties  to  these  fields.  We  begin  with  the  most 
relevant:  telerobotics  and  supervisory  control. 

7.1  Telerobotics  and  Teleoperation 

Sheridan’s  papers  and  books  on  telerobotics  and  supervisory  control 
are  perhaps  the  most  influential  in  the  field.  In  1992,  his  book  outlined 
the  state  of  the  art  in  human-robot  interaction,  with  an  emphasis  on 
open  problems,  mathematical  models,  and  information  flow  [267].  This 
book  was  followed  by  a  2002  updated  survey  and  framework  of  human 
factors  for  the  general  human-machine  interaction  problem  [268]. 

Even  more  influential  than  his  books,  perhaps,  is  Sheridan’s  and 
Verplank’s  levels  of  automation  in  human-machine  interaction.  These 
10  levels  of  automation  span  the  range  from  direct  control  through  deci¬ 
sion  support  to  supervisory  control  [269].  More  recently,  Parasuraman 
and  Wickens  teamed  with  Sheridan  to  extend  these  10  levels  of  automa¬ 
tion  beyond  decision  support  to  other  aspects  of  human-machine 
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interaction  [248].  Levels  of  automation  foreshadow  more  recent  con¬ 
cepts  of  dynamic  autonomy  in  all  its  forms  [67,  144]. 

In  addition  to  these  seminal  works,  there  are  numerous  examples 
of  remote  robot  operation.  One  example  comes  from  attempts  during 
World  War  II  to  remotely  control  aircraft.  This  lead  to  the  study  of 
remotely  piloted  vehicles  [95] ,  a  precursor  of  more  modern  work  on  the 
Human  Factors  of  Unmanned/Uninhabited  Aerial  Vehicles  [322]. 

Complementing  the  work  on  remotely  piloted  aircraft  is  a  work  on 
unmanned  underwater  vehicles  (UUVs).  This  work  includes  both  mil¬ 
itary  and  scientific  applications,  and  spans  topics  of  remote  visualiza¬ 
tion,  telepresence,  and  information  display  [329]. 

7.2  Human  Factors  and  Automation  Science 

The  field  of  human  factors  emerged  as  the  confluence  of  engineering 
psychology,  ergonomics,  and  accident  analysis.  Human  factors  work 
relevant  to  HRI  includes  important  lessons  from  thought  provoking 
papers  such  as  Bainbridge’s  “Ironies  of  Automation”  [14]  and  Hancock’s 
position  paper  on  the  make-up  of  HRI  teams  [121] .  Human  factors  work 
is  motivated  by  numerous  stories,  sometimes  humorous  and  sometimes 
sobering,  from  years  of  humans  interacting  with  automation  in  various 
forms  [50]. 

The  human  factors  literature  has  produced  key  concepts  of  inter¬ 
action,  such  as  mental  workload  [110,  204],  situation  awareness  [88], 
mental  models  [142,  300],  and  trust  in  automation  [173].  It  also  includes 
several  themes,  frameworks,  and  models  that  provide  a  solid  founda¬ 
tion  for  describing  and  predicting  responses  to  human-robot  interac¬ 
tion.  These  contributions  include  the  seminal  work  of  Rasmussen  who 
presented  a  hierarchy  of  interaction  that  included  knowledge-based, 
rule-based,  and  skill-based  interactions  [237].  Rasmussen’s  hierarchy 
is  a  human  factors  complement  to  hierarchical  and  intelligent  control 
[7,  183,  255].  Contributions  also  include  general  principles  of  cognitive 
ergonomics,  with  particularly  powerful  ideas  such  as  Wickens’s  Multi¬ 
ple  Resource  Theory  [314].  Complementing  these  models  are  interac¬ 
tion  phenomena  that  are  common  enough  that  they  are  elevated  to  the 
status  of  law  by  David  Woods  [320]. 
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Rich  as  these  models  and  laws  are,  they  cannot  substitute  for  prac¬ 
tical  real-world  observation.  This  point  was  strongly  made  by  Hutchins 
in  the  book  “Cognition  in  the  Wild”  [130].  In  the  spirit  of  real-world 
observation,  the  field  of  ethnography  has  developed  a  set  of  methodolo¬ 
gies  for  recording  observations  in  real-world  settings,  and  some  ethno¬ 
graphers  have  tried  to  translate  these  observation  and  summarization 
methods  into  tools  for  designing  interventions  [57,  58,  93]. 

Growing  out  of  the  need  to  understand  the  goals,  tasks,  and  informa¬ 
tion  flow  of  existing  processes,  a  series  of  methodologies  have  emerged 
that  produce  formalized  models  for  how  “things  get  done.”  These 
methodologies  include  Goal-Directed  Task  Analysis,  Cognitive  Task 
Analysis,  and  Cognitive  Work  Analysis  [88,  301].  These  methodologies 
produce  models  of  goals,  tasks,  and  information  flow,  which  are  being 
used  in  HRI  [4].  Complementing  these  high-level  models  are  cognitive 
models  of  the  mental  processes  used  to  accomplish  tasks,  and  activity 
analyses  of  existing  work  practices.  The  cognitive  models  and  activity 
analyses  are  especially  interesting  to  HRI  because  they  can  be  used 
not  only  as  models  of  existing  processes,  but  also  as  tools  to  generate 
behaviors  such  as  perspective-taking  and  planning  [295]. 

It  is  worth  noting  that  cognitive  psychology  and  social  psychology 
offer  perspectives  and  insights  that  are  distinct  from  traditional  human 
factors.  There  is  a  trend  in  HRI  to  include  cognitive  and  social  scien¬ 
tists  in  collaborative  research  efforts  with  roboticists,  human  factors 
engineers,  and  experts  in  human-computer  interaction. 

Given  the  rich  history  of  human  factors  and  the  recent  emergence  of 
HRI,  it  is  unfortunate  and  perhaps  inevitable  that  some  relevant  human 
factors  work  is  called  by  different  names  in  the  different  fields.  Exam¬ 
ples  include  adjustable  autonomy  and  Inagaki’s  Situation-Adaptive 
Autonomy  [132,  133];  and  augmented  reality /virtuality  and  synthetic 
vision  [48]. 

7.3  Aviation  and  Air  Traffic  Control 

Modern  aircraft  are  among  the  most  capable  semi-autonomous  systems 
in  use.  Moreover,  because  of  the  safety  critical  nature  of  aviation,  air¬ 
craft  systems  must  be  extremely  robust  and  reliable.  Careful  human 
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factors  analyses  are  often  performed  to  justify  a  change  to  an  aircraft 
system.  From  one  perspective,  an  aircraft  is  a  very  capable  type  of 
robot,  albeit  one  that  happens  to  carry  the  human  operator. 

As  a  result,  HRI  has  many  lessons  that  it  can  learn  from  aviation, 
both  in  terms  of  useful  technologies  and  careful  human  factors  analysis. 
Relevant  examples  include  ground  proximity  warning  systems,  which 
use  multi-modal  communications  coupled  with  robust  autonomy  to  pre¬ 
vent  controlled  flight  into  terrain  [63].  Tunnel-in-the-sky  displays  can 
increase  situation  awareness  by  helping  pilots  to  understand  how  con¬ 
trol  choices  will  affect  the  trajectory  of  the  aircraft  [195].  Problems 
caused  by  mode  confusion,  by  the  operator  being  out  of  the  loop,  by  vig¬ 
ilance,  by  excessive  workload,  and  by  team  coordination  issues  have  all 
received  attention  and  been  mitigated  by  procedures  and  technologies. 

As  robots  become  more  capable,  an  important  issue  is  how  many 
robots  can  be  managed  by  a  single  human.  This  question  makes  another 
aspect  of  aviation  relevant  to  HRI,  namely,  human  factors  work  done 
with  the  air  traffic  control  problem  (ATC).  ATC  is  a  problem  that 
involves  sequencing,  deconflicting,  and  handing  off  multiple  highly 
capable  systems  [123].  Indeed,  the  autonomy  level  of  these  aircraft 
is  extremely  high,  since  they  consist  of  both  a  trained  and  intelli¬ 
gent  human  operator  as  well  as  aircraft  autonomy.  Nevertheless,  ATC 
imposes  high  workloads  on  operators.  Careful  human  factors  analyses 
have  been  performed  and  mitigating  technologies  have  been  developed 
[86,  214],  Because  of  the  safety  critical  nature  of  ATC,  many  potentially 
useful  technologies  have  not  been  incorporated  into  ATC  systems.  Even 
so,  some  ATC-related  research  and  development  could  serve  as  a  type 
for  HRI  problems. 

There  are  three  other  aspects  of  aviation  and  ATC  that  are  very  rele¬ 
vant  to  HRI.  First,  ATC  training  and  certification  programs  have  many 
desirable  attributes  that  could  be  imitated  in  HRI.  Second,  because 
aviation  incidents  are  relatively  rare  and,  when  they  occur,  can  dam¬ 
age  career  prospects,  the  aviation  industry  has  developed  anonymous 
reporting  procedures  which  are  kept  in  a  database  of  problems  that 
have  occurred.  As  HRI  matures,  it  could  be  useful  to  create  a  standard¬ 
ized  reporting  system  to  identify  and  mitigate  problems  that  frequently 
arise.  Third,  the  aviation  industry  has  a  strong  set  of  standards.  There 
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have  been  recent  efforts  to  bring  the  standardization  process  to  HRI 
[92],  though  it  is  important  that  these  efforts  do  not  impose  undue 
restrictions  on  creativity  and  design. 

7.4  Intelligent  Vehicle  Systems 

The  field  of  intelligent  vehicle  systems  (IVS)  has  received  considerable 
attention  in  recent  decades,  including  the  emergence  of  several  confer¬ 
ences  and  journals  [1,  2,  3,  288].  The  field  of  intelligent  vehicle  systems 
has  many  problems  in  common  with  HRI,  including  designing  auton¬ 
omy  that  supports  human  behavior,  creating  attention-management 
aides,  supporting  planning  and  navigation  under  high- workload  con¬ 
ditions,  mitigating  errors,  and  creating  useful  models  and  metrics 
[27,  108,  109].  Indeed,  a  strong  case  can  be  made  that  modern  automo¬ 
biles  are  just  semi- autonomous  robots  that  carry  people. 

IVS  not  only  include  automobiles,  but  also  trains,  busses,  semi¬ 
trucks  and  other  forms  of  public  transit  [25].  The  users  of  IVS  range 
from  those  that  are  highly  trained  to  those  that  are  untrained  and 
sometimes  even  uninformed.  Moreover,  IVS  must  be  designed  to  be 
safety-critical,  time-critical,  and  to  operate  under  high  workload  con¬ 
ditions.  The  presence  of  untrained  operators  and  high-denrand  tasks 
produces  technologies  that  may  be  relevant  for  those  aspects  of  HRI 
that  require  interaction  with  bystanders  or  naive  operators. 

7.5  Human-Computer  Interaction  (HCI) 

As  the  field  of  HRI  has  grown,  it  has  seen  many  contributions  from 
researchers  in  HCI  and  it  has  been  nurtured  by  HCI  organizations.  For 
example,  the  first  International  Conference  on  Human-Robot  Inter¬ 
action  was  sponsored  by  ACM’s  Computer-Human  Interaction  Special 
Interest  Group  [265] .  HRI  research  is  attractive  to  many  members  of  the 
HCI  community  because  of  the  unique  challenges  posed  by  the  field. 
Of  particular  interest  is  the  fact  that  robots  occupy  physical  space. 
This  offers  unique  challenges  not  offered  in  desktop  metaphors  or  even 
pervasive  computing.  Physical  location  in  a  3D  space  imposes  strong 
requirements  on  how  information  is  displayed  in  remote  operation,  and 
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even  stronger  requirements  on  how  space  is  shared  when  robots  and 
humans  occupy  the  same  space.  HRI  benefits  from  contributions  from 
HCI  researchers,  both  in  methodologies,  design  principles,  and  com¬ 
puting  metaphors. 

7.6  Artificial  Intelligence  and  Cybernetics 

Because  of  their  emphasis  on  designing  intelligence  for  human-built 
systems,  the  fields  of  artificial  intelligence  (AI)  and  cybernetics  have  a 
great  deal  of  relevance  to  the  field  of  HRI.  Intelligence  and  autonomy 
are  closely  aligned.  Indeed,  when  experimenters  want  to  give  the  illusion 
of  truly  intelligent  robots,  it  is  common  to  use  a  “wizard  of  Oz”  design 
wherein  experiment  participants  believe  that  they  are  controlling  an 
intelligent  robot  but  where  in  reality  the  commands  that  they  issue 
are  received  and  translated  into  teleoperation  commands  by  a  hidden 
human  [116]. 

HRI  frequently  uses  concepts  from  AI  in  the  design  of  autonomy 
algorithms.  Moreover,  AI  techniques  have  informed  and  been  informed 
by  concepts  from  cognitive  science.  For  example,  the  ACT-R  system, 
a  popular  tool  for  modeling  cognition,  uses  Al-like  production  rules. 
Such  cognitive  models  have  increasingly  become  relevant  to  HRI,  both 
as  tools  for  modeling  how  a  human  might  interact  and  as  the  basis  for 
generating  robot  behavior  [295]. 

Although  sometimes  justifiably  treated  as  a  separate  field  from  AI, 
augmented  reality  and  telepresence  have  much  relevance  to  HRI.  Aug¬ 
mented  reality  techniques  are  used  to  support  remote  interactions  in 
NASA’s  Robonaut  [9].  Augmented  virtuality  and  mixed  reality  are  vari¬ 
ations  of  augmented  reality  that  have  found  application  in  HRI  [208]. 
Some  suggest  that  telepresence,  the  natural  extension  of  human  aware¬ 
ness  of  a  remote  space,  is  a  goal  of  interface  design  in  HRI,  though 
others  note  that  a  feeling  of  remote  presence  is  not  necessary  provided 
that  information  is  displayed  in  a  way  that  supports  intentional  action 
in  the  remote  space  [208] . 

Another  Al-related  area  that  has  developed  into  a  separate  field 
of  study  is  computer  vision.  Computer  vision  algorithms  are  frequently 
used  to  translate  camera  imagery  into  percepts  that  support  autonomy. 
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Moreover,  these  algorithms  are  also  used  to  provide  enhanced  aware¬ 
ness  of  information  through  the  use  of  image  stabilization,  mosaics, 
automated  target  recognition,  and  image  enhancement. 

Many  AI  techniques  are  used  in  computer  games.  These  games, 
some  of  which  are  very  sophisticated,  provide  a  probe  into  the  levels  of 
autonomy  needed  to  support  useful  interactions.  Given  these  levels  of 
autonomy,  information  is  integrated  and  presented  to  operators  in  sev¬ 
eral  different  forms;  evaluating  these  forms  of  information  presentation 
provides  guidelines  for  interface  designers  in  HRI  [241].  Sophisticated 
multi-player  online  games  may  become  useful  in  understanding  how 
natural  language  can  be  used  to  support  HRI  and  how  human-robot 
teams  should  interact. 

Finally,  machine  learning  is  a  subfield  of  AI  that  is  proving  very 
useful  in  robotics  and  HRI.  Machine  learning  can  be  used  to  develop 
robot  behaviors,  robot  perception,  and  multi-robot  interaction  [40,  89, 
207].  Interactive  learning  has  received  attention  as  a  way  to  capture 
and  encode  useful  robot  behaviors,  to  provide  robot  training,  and  to 
improve  perception.  Interactive  techniques  with  intelligent  systems  are 
also  present  in  AI.  Interactive  proof  system,  interactive  planners,  and 
“programming  by  reward”  in  machine  learning  are  all  examples  of  how 
human  input  can  be  used  in  collaboration  with  AI  algorithms. 

7.7  Haptics  and  Telemanipulation 

Before  concluding  the  review,  it  is  important  to  note  that  much  of 
the  field  of  haptics  and  telemanipulation  are  aligned  with  the  goals 
and  challenge  problems  of  HRI.  However,  the  current  research  culture 
tends  to  treat  haptics/telemanipulation  separate  from  HRI,  perhaps 
because  of  the  longer  history  of  the  field  of  haptics.  Since  the  two  fields 
have  much  to  learn  from  each  other,  it  is  desirable  that  the  research 
communities  increase  interactions. 
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Summary 


Human-robot  interaction  is  a  growing  field  of  research  and  application. 
The  field  includes  many  challenging  problems  and  has  the  potential 
to  produce  solutions  with  positive  social  impact.  Its  interdisciplinary 
nature  requires  that  researchers  in  the  field  understand  their  research 
within  a  broader  context.  In  this  survey,  we  have  tried  to  present  a 
unified  treatment  of  HRI-related  problems,  identify  key  themes,  and 
discuss  challenge  problems  that  are  likely  to  shape  the  field  in  the  near 
future. 
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